2024-04-22T06:20:05.448 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:20:05.448 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130 --name leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi --verbose -- /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/orig.config.yaml 2024-04-22T06:20:05.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:20:05.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} email: null first_in_suite: false job_id: '7668130' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4362 sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 sleep_before_teardown: 0 subset: 222/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPA11MmHfT2Cz2yueIY3/AZ3HnxXL90Bx+Xt2XD5gaIbkpiWrejI8JgWy7Lkcj5rKE+FV7D9iNyy3TLW99f1a4g= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVZEhP1y27H0IGqMvfG9b1BSfrpzUiVOVMTvUFxg1JgX0Ki8vRk2Y+fYFd6u6QtVTA6LDlpM0QLQpZq66ocqBQ= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/cWEzCQzEc16LopmxSMc5SWmlLThkApqEjNKRAO+/sy6QsDmmTiqBJiYPlCrd1ptCnj881sJuxwlAyLkQen74= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:52:53 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:20:05.606 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa; will attempt to use it 2024-04-22T06:20:05.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks 2024-04-22T06:20:05.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:20:05.608 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:20:05.637 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '86f7587a5a09af35f0895e1a2d08527638fae697' 2024-04-22T06:20:05.637 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:20:05.638 INFO:teuthology.packaging:ref: None 2024-04-22T06:20:05.638 INFO:teuthology.packaging:tag: None 2024-04-22T06:20:05.638 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-22T06:20:05.638 INFO:teuthology.packaging:sha1: 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:20:05.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-22T06:20:05.818 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.g86f7587a 2024-04-22T06:20:05.820 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:20:05.832 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:20:05.832 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:20:05.910 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:20:05.935 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:20:05.946 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:20:05.974 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.830935', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPA11MmHfT2Cz2yueIY3/AZ3HnxXL90Bx+Xt2XD5gaIbkpiWrejI8JgWy7Lkcj5rKE+FV7D9iNyy3TLW99f1a4g='} 2024-04-22T06:20:06.013 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.833474', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVZEhP1y27H0IGqMvfG9b1BSfrpzUiVOVMTvUFxg1JgX0Ki8vRk2Y+fYFd6u6QtVTA6LDlpM0QLQpZq66ocqBQ='} 2024-04-22T06:20:06.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.832193', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/cWEzCQzEc16LopmxSMc5SWmlLThkApqEjNKRAO+/sy6QsDmmTiqBJiYPlCrd1ptCnj881sJuxwlAyLkQen74='} 2024-04-22T06:20:06.039 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:20:06.051 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-22T06:20:06.051 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-22T06:20:06.051 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-22T06:20:06.051 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:20:06.224 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f691b50f0d0>, signals=[15]) 2024-04-22T06:20:06.224 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:20:06.235 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:20:06.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:20:06.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:20:06.325 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-04-22T06:20:06.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:20:06.408 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-04-22T06:20:06.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:20:06.506 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:20:06.512 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-04-22T06:20:06.530 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-04-22T06:20:06.531 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-22T06:20:06.587 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:20:06.588 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:20:06.589 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-22T06:20:06.613 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-04-22T06:20:06.631 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-04-22T06:20:06.631 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:20:06.688 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:20:06.689 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:20:06.689 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:20:06.689 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:20:06.689 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:20:06.689 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-04-22T06:20:06.713 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-04-22T06:20:06.731 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-04-22T06:20:06.731 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-04-22T06:20:06.788 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:20:06.789 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:20:06.790 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-04-22T06:20:06.790 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:20:06.790 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:20:06.790 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:20:06.790 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:20:06.791 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:20:06.791 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:20:06.791 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-04-22T06:20:06.822 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:20:06.840 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:20:06.851 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:20:06.851 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:20:06.854 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:20:06.857 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:20:06.873 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:20:06.884 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:20:06.884 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:20:06.911 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:20:06.913 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:20:07.003 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:20:07.120 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:20:07.142 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-22T06:20:07.142 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-22T06:20:07.142 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-22T06:20:07.143 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:20:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:07.160 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-04-22T06:20:07.217 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:20:07.217 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-437.el9.x86_64 2024-04-22T06:20:07.217 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-04-22T06:20:09.406 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-22T06:20:13.199 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-04-22T06:20:17.316 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-22T06:20:19.091 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-22T06:20:20.618 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-22T06:20:25.517 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-22T06:20:27.373 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:20:27.374 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:20:27.464 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-22T06:20:27.465 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-22T06:20:27.465 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-22T06:20:27.574 DEBUG:teuthology.orchestra.run.smithi105:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:20:28.073 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:20:25 AM UTC. 2024-04-22T06:20:28.209 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-22T06:20:28.219 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.219 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-22T06:20:28.219 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout:Reinstalling: 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 255 k 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 0 2024-04-22T06:20:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:Operation aborted. 2024-04-22T06:20:28.272 INFO:teuthology.orchestra.run.smithi105.stdout:Is this ok [y/N]: 2024-04-22T06:20:28.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum reinstall -y kernel || true 2024-04-22T06:20:28.760 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:20:25 AM UTC. 2024-04-22T06:20:28.878 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-22T06:20:28.879 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.879 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-22T06:20:28.879 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.879 INFO:teuthology.orchestra.run.smithi105.stdout:Reinstalling: 2024-04-22T06:20:28.879 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 255 k 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 0 2024-04-22T06:20:28.880 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-22T06:20:30.178 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-437.el9.x86_64.rpm 248 kB/s | 255 kB 00:01 2024-04-22T06:20:30.178 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:20:30.179 INFO:teuthology.orchestra.run.smithi105.stdout:Total 196 kB/s | 255 kB 00:01 2024-04-22T06:20:30.194 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-22T06:20:30.360 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-22T06:20:30.360 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-22T06:20:30.365 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-22T06:20:30.365 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-22T06:20:30.499 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-22T06:20:30.562 INFO:teuthology.orchestra.run.smithi105.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:20:30.563 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:20:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:20:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:20:31.882 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:20:31.883 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:31.883 INFO:teuthology.orchestra.run.smithi105.stdout:Reinstalled: 2024-04-22T06:20:31.883 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:20:31.883 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:20:31.883 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-22T06:20:32.047 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:20:32.094 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:20:32.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:20:32.094 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:20:32.094 DEBUG:teuthology.parallel:result is None 2024-04-22T06:20:32.094 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:20:32.103 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:20:32.103 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:20:32.106 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:20:32.108 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:20:32.126 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:20:32.164 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:20:32.189 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:20:32.189 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:20:32.193 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:20:32.195 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:20:32.318 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:20:32.326 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:20:32.327 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:20:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:32.343 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:20:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:32.360 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:20:32.363 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:20:32.386 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:20:32.389 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.408 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.454 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.462 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.470 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.474 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:20:32.476 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:20:32.488 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:20:32.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:20:32.490 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:20:32.493 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:20:32.519 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:20:32.559 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:20:32.559 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:20:32.561 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:20:32.564 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:20:32.580 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:32.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:32.699 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:32.792 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:32.819 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:20:32.820 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:20:32.883 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:32.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:32.976 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:33.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:33.102 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:20:33.102 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:20:33.168 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:33.214 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:20:33.262 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:33.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:20:33.386 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:20:33.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:20:33.452 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-04-22T06:20:33.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-04-22T06:20:33.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-04-22T06:20:33.488 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:20:33.492 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:20:33.534 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:20:34.038 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:20:34.046 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:20:34.046 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:20:34.095 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:20:34.131 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:20:34.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-04-22T06:20:34.195 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-04-22T06:20:34.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-04-22T06:20:34.257 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-04-22T06:20:34.259 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-04-22T06:20:34.306 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-04-22T06:20:34.307 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:20:34.308 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-04-22T06:20:34.329 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-04-22T06:20:34.329 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-04-22T06:20:34.359 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-04-22T06:20:34.359 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-04-22T06:20:34.387 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-04-22T06:20:34.387 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:20:34.388 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:20:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:34.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:20:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:34.442 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:20:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:34.470 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:20:34.470 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:20:34.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:20:34.523 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:20:34.550 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:20:34.560 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-22T06:20:34.566 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-22T06:20:34.783 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:20:34.812 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:20:34.813 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com,smithi112.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:24:23.793 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-04-22T06:24:23.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-22T06:24:23.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:24:23.878 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-22T06:24:23.957 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-22T06:24:23.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-04-22T06:24:23.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:24:24.058 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-04-22T06:24:24.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-04-22T06:24:24.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-22T06:24:24.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:24:24.222 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-04-22T06:24:24.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-04-22T06:24:24.301 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:24:24.313 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:24:24.313 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:24:24.314 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:24:24.316 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:24:24.316 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:24:24.318 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:24:24.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:24:24.342 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:24:24.347 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:24:24.355 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:24:24.362 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:24:24.370 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:24:24.378 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-04-22T06:24:24.384 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:24:24.387 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-04-22T06:24:24.388 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-04-22T06:24:24.397 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-04-22T06:24:24.400 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:24:24.407 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-04-22T06:24:24.412 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:24:24.413 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:24:24.418 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-04-22T06:24:24.425 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:24:24.431 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:24:24.444 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:24:24.473 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:24:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:24:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-22T06:24:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.477 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.477 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.477 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:24:24.480 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:24:24.480 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-04-22T06:24:24.480 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.480 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.481 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.481 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.500 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.504 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:24:24.505 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:24:24.513 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:24:24.513 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'}} 2024-04-22T06:24:24.513 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697'} 2024-04-22T06:24:24.513 INFO:teuthology.task.install:Using flavor: default 2024-04-22T06:24:24.520 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:24:24.520 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-22T06:24:24.520 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:24:24.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:24:24.523 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:24:24.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:24:24.524 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'tag': None, 'wait_for_package': False} 2024-04-22T06:24:24.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:24:24.684 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:24:24.685 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:24:24.763 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:24:24.763 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:24:24.775 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/ 2024-04-22T06:24:24.775 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.g86f7587a 2024-04-22T06:24:24.886 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:24:24.886 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:24:24.886 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:24:24.890 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:24:24.891 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:24:24.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:24:24.911 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:24:24.911 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:24:24.915 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:24:24.915 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:24:24.957 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:24:24.957 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:24:24.957 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:24:24.976 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:24:24.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:24:24.982 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-22T06:24:24.982 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/86f7587a5a09af35f0895e1a2d08527638fae697/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:24:25.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:24:25.049 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:24:25.055 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:24:25.112 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-04-22T06:24:25.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-04-22T06:24:25.117 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-04-22T06:24:25.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-04-22T06:24:25.120 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:24:25.183 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-04-22T06:24:25.184 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-04-22T06:24:25.335 INFO:teuthology.orchestra.run.smithi105.stdout:48 files removed 2024-04-22T06:24:25.341 INFO:teuthology.orchestra.run.smithi112.stdout:48 files removed 2024-04-22T06:24:25.362 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:24:25.368 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:24:25.407 INFO:teuthology.orchestra.run.smithi177.stdout:48 files removed 2024-04-22T06:24:25.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-22T06:24:26.017 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 213 kB/s | 83 kB 00:00 2024-04-22T06:24:26.030 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-04-22T06:24:26.094 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-04-22T06:24:26.376 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-22T06:24:26.406 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-22T06:24:26.545 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-22T06:24:26.726 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-22T06:24:26.756 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-22T06:24:26.987 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-22T06:24:27.418 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-22T06:24:28.021 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 8.0 MB/s | 8.0 MB 00:00 2024-04-22T06:24:28.415 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-04-22T06:24:29.778 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-22T06:24:30.332 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-22T06:24:32.222 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-04-22T06:24:34.020 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-04-22T06:24:34.484 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-22T06:24:35.737 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-22T06:24:36.205 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-22T06:24:36.409 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-04-22T06:24:37.823 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-22T06:24:39.541 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-22T06:24:39.619 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-04-22T06:24:44.449 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-22T06:24:44.686 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-22T06:24:46.516 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.517 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.527 INFO:teuthology.orchestra.run.smithi177.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:24:46.528 INFO:teuthology.orchestra.run.smithi177.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.529 INFO:teuthology.orchestra.run.smithi177.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:24:46.530 INFO:teuthology.orchestra.run.smithi177.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:24:46.531 INFO:teuthology.orchestra.run.smithi177.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:24:46.534 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:24:46.535 INFO:teuthology.orchestra.run.smithi177.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:24:46.593 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-04-22T06:24:46.613 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:24:46.614 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:24:46.615 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:24:46.616 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:24:46.617 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-04-22T06:24:46.618 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:24:46.618 INFO:teuthology.orchestra.run.smithi177.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:24:46.618 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:24:46.618 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:24:46.618 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:24:46.619 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:24:46.620 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:24:46.620 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:24:46.620 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:24:46.620 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:24:46.620 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:24:46.621 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:24:46.622 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:24:46.623 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:24:46.624 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:24:46.624 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:24:46.624 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:24:46.624 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:24:46.624 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:24:46.625 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:24:46.626 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:24:46.627 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:24:46.628 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:24:46.629 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:24:46.630 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:24:46.631 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:24:46.632 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout:Install 148 Packages 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-04-22T06:24:46.633 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:24:46.634 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 218 M 2024-04-22T06:24:46.634 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-04-22T06:24:46.787 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.787 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.798 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:24:46.799 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:24:46.800 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:24:46.801 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:24:46.802 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:24:46.805 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:24:46.806 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:24:46.865 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-22T06:24:46.874 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-22T06:24:46.874 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-22T06:24:46.874 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-22T06:24:46.874 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-22T06:24:46.874 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:24:46.875 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:24:46.876 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:24:46.877 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:24:46.878 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:24:46.878 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-04-22T06:24:46.878 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:24:46.879 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:24:46.879 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-22T06:24:46.879 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:24:46.879 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:24:46.880 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:24:46.881 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:24:46.882 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:24:46.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:24:46.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:24:46.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:24:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:24:46.887 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout:Install 148 Packages 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-04-22T06:24:46.888 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:24:46.889 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 218 M 2024-04-22T06:24:46.889 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-22T06:24:47.451 INFO:teuthology.orchestra.run.smithi177.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-04-22T06:24:47.667 INFO:teuthology.orchestra.run.smithi177.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 4.0 MB/s | 890 kB 00:00 2024-04-22T06:24:47.728 INFO:teuthology.orchestra.run.smithi105.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-22T06:24:47.734 INFO:teuthology.orchestra.run.smithi177.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-22T06:24:48.084 INFO:teuthology.orchestra.run.smithi177.stdout:(4/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 6.7 MB/s | 2.4 MB 00:00 2024-04-22T06:24:48.401 INFO:teuthology.orchestra.run.smithi177.stdout:(5/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-04-22T06:24:48.485 INFO:teuthology.orchestra.run.smithi177.stdout:(6/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 4.5 MB/s | 5.4 MB 00:01 2024-04-22T06:24:48.662 INFO:teuthology.orchestra.run.smithi105.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 954 kB/s | 890 kB 00:00 2024-04-22T06:24:48.854 INFO:teuthology.orchestra.run.smithi105.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 756 kB/s | 145 kB 00:00 2024-04-22T06:24:49.121 INFO:teuthology.orchestra.run.smithi105.stdout:(4/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 3.3 MB/s | 5.4 MB 00:01 2024-04-22T06:24:49.471 INFO:teuthology.orchestra.run.smithi105.stdout:(5/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 4.3 MB/s | 1.5 MB 00:00 2024-04-22T06:24:49.518 INFO:teuthology.orchestra.run.smithi177.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 4.2 MB/s | 4.7 MB 00:01 2024-04-22T06:24:49.852 INFO:teuthology.orchestra.run.smithi177.stdout:(8/150): ceph-common-19.0.0-3156.g86f7587a.el9. 8.2 MB/s | 21 MB 00:02 2024-04-22T06:24:49.927 INFO:teuthology.orchestra.run.smithi177.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 336 kB/s | 25 kB 00:00 2024-04-22T06:24:50.546 INFO:teuthology.orchestra.run.smithi105.stdout:(6/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-04-22T06:24:50.604 INFO:teuthology.orchestra.run.smithi105.stdout:(7/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 1.3 MB/s | 2.4 MB 00:01 2024-04-22T06:24:50.922 INFO:teuthology.orchestra.run.smithi105.stdout:(8/150): ceph-common-19.0.0-3156.g86f7587a.el9. 6.1 MB/s | 21 MB 00:03 2024-04-22T06:24:50.988 INFO:teuthology.orchestra.run.smithi105.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 380 kB/s | 25 kB 00:00 2024-04-22T06:24:52.227 INFO:teuthology.orchestra.run.smithi177.stdout:(10/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 3.9 MB/s | 11 MB 00:02 2024-04-22T06:24:52.394 INFO:teuthology.orchestra.run.smithi177.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 1.3 MB/s | 225 kB 00:00 2024-04-22T06:24:52.486 INFO:teuthology.orchestra.run.smithi177.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 295 kB/s | 27 kB 00:00 2024-04-22T06:24:52.578 INFO:teuthology.orchestra.run.smithi177.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 346 kB/s | 32 kB 00:00 2024-04-22T06:24:52.845 INFO:teuthology.orchestra.run.smithi177.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 2.7 MB/s | 747 kB 00:00 2024-04-22T06:24:52.936 INFO:teuthology.orchestra.run.smithi177.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 1.7 MB/s | 164 kB 00:00 2024-04-22T06:24:53.037 INFO:teuthology.orchestra.run.smithi177.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 1.2 MB/s | 127 kB 00:00 2024-04-22T06:24:53.212 INFO:teuthology.orchestra.run.smithi177.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 2.8 MB/s | 500 kB 00:00 2024-04-22T06:24:53.662 INFO:teuthology.orchestra.run.smithi177.stdout:(18/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 3.3 MB/s | 17 MB 00:05 2024-04-22T06:24:53.804 INFO:teuthology.orchestra.run.smithi177.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 319 kB/s | 45 kB 00:00 2024-04-22T06:24:53.946 INFO:teuthology.orchestra.run.smithi177.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 951 kB/s | 135 kB 00:00 2024-04-22T06:24:54.063 INFO:teuthology.orchestra.run.smithi177.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 1.4 MB/s | 162 kB 00:00 2024-04-22T06:24:54.280 INFO:teuthology.orchestra.run.smithi177.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 1.4 MB/s | 320 kB 00:00 2024-04-22T06:24:54.455 INFO:teuthology.orchestra.run.smithi177.stdout:(23/150): python3-rbd-19.0.0-3156.g86f7587a.el9 1.7 MB/s | 298 kB 00:00 2024-04-22T06:24:54.564 INFO:teuthology.orchestra.run.smithi177.stdout:(24/150): python3-rgw-19.0.0-3156.g86f7587a.el9 918 kB/s | 100 kB 00:00 2024-04-22T06:24:54.639 INFO:teuthology.orchestra.run.smithi177.stdout:(25/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 3.8 MB/s | 5.4 MB 00:01 2024-04-22T06:24:54.664 INFO:teuthology.orchestra.run.smithi177.stdout:(26/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 853 kB/s | 85 kB 00:00 2024-04-22T06:24:54.765 INFO:teuthology.orchestra.run.smithi177.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-22T06:24:54.822 INFO:teuthology.orchestra.run.smithi105.stdout:(10/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 3.9 MB/s | 17 MB 00:04 2024-04-22T06:24:54.840 INFO:teuthology.orchestra.run.smithi177.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 457 kB/s | 34 kB 00:00 2024-04-22T06:24:54.905 INFO:teuthology.orchestra.run.smithi105.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 2.6 MB/s | 225 kB 00:00 2024-04-22T06:24:54.907 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-04-22T06:24:54.958 INFO:teuthology.orchestra.run.smithi177.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 1.2 MB/s | 142 kB 00:00 2024-04-22T06:24:54.980 INFO:teuthology.orchestra.run.smithi105.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 360 kB/s | 27 kB 00:00 2024-04-22T06:24:55.064 INFO:teuthology.orchestra.run.smithi105.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 381 kB/s | 32 kB 00:00 2024-04-22T06:24:55.233 INFO:teuthology.orchestra.run.smithi177.stdout:(30/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 5.2 MB/s | 3.1 MB 00:00 2024-04-22T06:24:55.273 INFO:teuthology.orchestra.run.smithi105.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 3.5 MB/s | 747 kB 00:00 2024-04-22T06:24:55.365 INFO:teuthology.orchestra.run.smithi105.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 1.7 MB/s | 164 kB 00:00 2024-04-22T06:24:55.448 INFO:teuthology.orchestra.run.smithi105.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 1.5 MB/s | 127 kB 00:00 2024-04-22T06:24:55.967 INFO:teuthology.orchestra.run.smithi177.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 3.5 MB/s | 3.5 MB 00:01 2024-04-22T06:24:56.007 INFO:teuthology.orchestra.run.smithi105.stdout:(17/150): ceph-test-19.0.0-3156.g86f7587a.el9.x 9.8 MB/s | 49 MB 00:05 2024-04-22T06:24:56.032 INFO:teuthology.orchestra.run.smithi105.stdout:(18/150): libradosstriper1-19.0.0-3156.g86f7587 857 kB/s | 500 kB 00:00 2024-04-22T06:24:56.042 INFO:teuthology.orchestra.run.smithi177.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3156.g86 3.2 MB/s | 249 kB 00:00 2024-04-22T06:24:56.082 INFO:teuthology.orchestra.run.smithi105.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 903 kB/s | 45 kB 00:00 2024-04-22T06:24:56.141 INFO:teuthology.orchestra.run.smithi105.stdout:(20/150): python3-ceph-common-19.0.0-3156.g86f7 2.3 MB/s | 135 kB 00:00 2024-04-22T06:24:56.200 INFO:teuthology.orchestra.run.smithi105.stdout:(21/150): python3-cephfs-19.0.0-3156.g86f7587a. 2.7 MB/s | 162 kB 00:00 2024-04-22T06:24:56.300 INFO:teuthology.orchestra.run.smithi105.stdout:(22/150): python3-rados-19.0.0-3156.g86f7587a.e 3.1 MB/s | 320 kB 00:00 2024-04-22T06:24:56.409 INFO:teuthology.orchestra.run.smithi105.stdout:(23/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 13 MB/s | 5.4 MB 00:00 2024-04-22T06:24:56.434 INFO:teuthology.orchestra.run.smithi105.stdout:(24/150): python3-rbd-19.0.0-3156.g86f7587a.el9 2.2 MB/s | 298 kB 00:00 2024-04-22T06:24:56.459 INFO:teuthology.orchestra.run.smithi105.stdout:(25/150): python3-rgw-19.0.0-3156.g86f7587a.el9 1.9 MB/s | 100 kB 00:00 2024-04-22T06:24:56.484 INFO:teuthology.orchestra.run.smithi177.stdout:(33/150): ceph-test-19.0.0-3156.g86f7587a.el9.x 7.5 MB/s | 49 MB 00:06 2024-04-22T06:24:56.484 INFO:teuthology.orchestra.run.smithi105.stdout:(26/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-22T06:24:56.509 INFO:teuthology.orchestra.run.smithi177.stdout:(34/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 106 kB/s | 49 kB 00:00 2024-04-22T06:24:56.535 INFO:teuthology.orchestra.run.smithi177.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3156.g8 329 kB/s | 16 kB 00:00 2024-04-22T06:24:56.543 INFO:teuthology.orchestra.run.smithi105.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-22T06:24:56.601 INFO:teuthology.orchestra.run.smithi105.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 588 kB/s | 34 kB 00:00 2024-04-22T06:24:56.618 INFO:teuthology.orchestra.run.smithi177.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-22T06:24:56.660 INFO:teuthology.orchestra.run.smithi177.stdout:(37/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.8 MB/s | 273 kB 00:00 2024-04-22T06:24:56.685 INFO:teuthology.orchestra.run.smithi177.stdout:(38/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 274 kB/s | 18 kB 00:00 2024-04-22T06:24:56.694 INFO:teuthology.orchestra.run.smithi105.stdout:(29/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 13 MB/s | 3.1 MB 00:00 2024-04-22T06:24:56.710 INFO:teuthology.orchestra.run.smithi177.stdout:(39/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 4.2 MB/s | 763 kB 00:00 2024-04-22T06:24:56.719 INFO:teuthology.orchestra.run.smithi105.stdout:(30/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 1.2 MB/s | 142 kB 00:00 2024-04-22T06:24:56.794 INFO:teuthology.orchestra.run.smithi177.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 600 kB/s | 80 kB 00:00 2024-04-22T06:24:56.870 INFO:teuthology.orchestra.run.smithi177.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-22T06:24:56.895 INFO:teuthology.orchestra.run.smithi177.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-22T06:24:56.920 INFO:teuthology.orchestra.run.smithi105.stdout:(31/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 1.7 MB/s | 11 MB 00:06 2024-04-22T06:24:56.920 INFO:teuthology.orchestra.run.smithi177.stdout:(43/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-04-22T06:24:56.970 INFO:teuthology.orchestra.run.smithi177.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-22T06:24:56.978 INFO:teuthology.orchestra.run.smithi105.stdout:(32/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 12 MB/s | 3.5 MB 00:00 2024-04-22T06:24:57.004 INFO:teuthology.orchestra.run.smithi177.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-22T06:24:57.028 INFO:teuthology.orchestra.run.smithi105.stdout:(33/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 993 kB/s | 49 kB 00:00 2024-04-22T06:24:57.029 INFO:teuthology.orchestra.run.smithi177.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-22T06:24:57.054 INFO:teuthology.orchestra.run.smithi105.stdout:(34/150): ceph-mgr-modules-core-19.0.0-3156.g86 1.8 MB/s | 249 kB 00:00 2024-04-22T06:24:57.054 INFO:teuthology.orchestra.run.smithi177.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-22T06:24:57.079 INFO:teuthology.orchestra.run.smithi105.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3156.g8 328 kB/s | 16 kB 00:00 2024-04-22T06:24:57.080 INFO:teuthology.orchestra.run.smithi177.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-22T06:24:57.179 INFO:teuthology.orchestra.run.smithi105.stdout:(36/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 7.5 MB/s | 763 kB 00:00 2024-04-22T06:24:57.188 INFO:teuthology.orchestra.run.smithi177.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 988 kB/s | 107 kB 00:00 2024-04-22T06:24:57.204 INFO:teuthology.orchestra.run.smithi105.stdout:(37/150): ceph-volume-19.0.0-3156.g86f7587a.el9 1.8 MB/s | 273 kB 00:00 2024-04-22T06:24:57.230 INFO:teuthology.orchestra.run.smithi177.stdout:(50/150): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-22T06:24:57.232 INFO:teuthology.orchestra.run.smithi105.stdout:(38/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 348 kB/s | 18 kB 00:00 2024-04-22T06:24:57.255 INFO:teuthology.orchestra.run.smithi177.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-22T06:24:57.297 INFO:teuthology.orchestra.run.smithi177.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-04-22T06:24:57.322 INFO:teuthology.orchestra.run.smithi177.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-04-22T06:24:57.340 INFO:teuthology.orchestra.run.smithi105.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 593 kB/s | 80 kB 00:00 2024-04-22T06:24:57.356 INFO:teuthology.orchestra.run.smithi177.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-04-22T06:24:57.382 INFO:teuthology.orchestra.run.smithi177.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 309 kB/s | 18 kB 00:00 2024-04-22T06:24:57.407 INFO:teuthology.orchestra.run.smithi177.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-22T06:24:57.407 INFO:teuthology.orchestra.run.smithi105.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-22T06:24:57.432 INFO:teuthology.orchestra.run.smithi177.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-22T06:24:57.441 INFO:teuthology.orchestra.run.smithi105.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-22T06:24:57.457 INFO:teuthology.orchestra.run.smithi177.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-04-22T06:24:57.466 INFO:teuthology.orchestra.run.smithi105.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-22T06:24:57.491 INFO:teuthology.orchestra.run.smithi177.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-22T06:24:57.517 INFO:teuthology.orchestra.run.smithi105.stdout:(43/150): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-22T06:24:57.542 INFO:teuthology.orchestra.run.smithi105.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-22T06:24:57.550 INFO:teuthology.orchestra.run.smithi177.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-22T06:24:57.567 INFO:teuthology.orchestra.run.smithi105.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-22T06:24:57.575 INFO:teuthology.orchestra.run.smithi177.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-04-22T06:24:57.592 INFO:teuthology.orchestra.run.smithi105.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-22T06:24:57.600 INFO:teuthology.orchestra.run.smithi177.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-22T06:24:57.617 INFO:teuthology.orchestra.run.smithi105.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-22T06:24:57.625 INFO:teuthology.orchestra.run.smithi177.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-22T06:24:57.651 INFO:teuthology.orchestra.run.smithi177.stdout:(64/150): libtool-2.4.6-45.el9.x86_64.rpm 2.9 MB/s | 582 kB 00:00 2024-04-22T06:24:57.676 INFO:teuthology.orchestra.run.smithi177.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 700 kB/s | 35 kB 00:00 2024-04-22T06:24:57.702 INFO:teuthology.orchestra.run.smithi177.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-22T06:24:57.709 INFO:teuthology.orchestra.run.smithi105.stdout:(48/150): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-04-22T06:24:57.760 INFO:teuthology.orchestra.run.smithi177.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-04-22T06:24:57.819 INFO:teuthology.orchestra.run.smithi177.stdout:(68/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 29 MB/s | 4.1 MB 00:00 2024-04-22T06:24:57.885 INFO:teuthology.orchestra.run.smithi105.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 611 kB/s | 107 kB 00:00 2024-04-22T06:24:57.907 INFO:teuthology.orchestra.run.smithi177.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-04-22T06:24:57.936 INFO:teuthology.orchestra.run.smithi177.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-04-22T06:24:57.943 INFO:teuthology.orchestra.run.smithi105.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-22T06:24:57.969 INFO:teuthology.orchestra.run.smithi105.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-04-22T06:24:57.969 INFO:teuthology.orchestra.run.smithi177.stdout:(71/150): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-04-22T06:24:58.002 INFO:teuthology.orchestra.run.smithi105.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-04-22T06:24:58.011 INFO:teuthology.orchestra.run.smithi177.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-22T06:24:58.027 INFO:teuthology.orchestra.run.smithi105.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-04-22T06:24:58.036 INFO:teuthology.orchestra.run.smithi177.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-22T06:24:58.061 INFO:teuthology.orchestra.run.smithi105.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-22T06:24:58.079 INFO:teuthology.orchestra.run.smithi177.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-04-22T06:24:58.086 INFO:teuthology.orchestra.run.smithi105.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-22T06:24:58.120 INFO:teuthology.orchestra.run.smithi105.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-22T06:24:58.145 INFO:teuthology.orchestra.run.smithi105.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-22T06:24:58.154 INFO:teuthology.orchestra.run.smithi177.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-22T06:24:58.179 INFO:teuthology.orchestra.run.smithi105.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 773 kB/s | 45 kB 00:00 2024-04-22T06:24:58.182 INFO:teuthology.orchestra.run.smithi177.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-22T06:24:58.196 INFO:teuthology.orchestra.run.smithi177.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 613 kB/s | 26 kB 00:00 2024-04-22T06:24:58.213 INFO:teuthology.orchestra.run.smithi177.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 574 kB/s | 19 kB 00:00 2024-04-22T06:24:58.238 INFO:teuthology.orchestra.run.smithi177.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-22T06:24:58.255 INFO:teuthology.orchestra.run.smithi177.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 829 kB/s | 35 kB 00:00 2024-04-22T06:24:58.271 INFO:teuthology.orchestra.run.smithi105.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-22T06:24:58.313 INFO:teuthology.orchestra.run.smithi105.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 4.3 MB/s | 582 kB 00:00 2024-04-22T06:24:58.330 INFO:teuthology.orchestra.run.smithi177.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-04-22T06:24:58.338 INFO:teuthology.orchestra.run.smithi105.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-04-22T06:24:58.356 INFO:teuthology.orchestra.run.smithi177.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-22T06:24:58.371 INFO:teuthology.orchestra.run.smithi105.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-04-22T06:24:58.381 INFO:teuthology.orchestra.run.smithi177.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-22T06:24:58.407 INFO:teuthology.orchestra.run.smithi177.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-22T06:24:58.438 INFO:teuthology.orchestra.run.smithi105.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.0 MB/s | 3.0 MB 00:00 2024-04-22T06:24:58.438 INFO:teuthology.orchestra.run.smithi177.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 838 kB/s | 42 kB 00:00 2024-04-22T06:24:58.455 INFO:teuthology.orchestra.run.smithi105.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-22T06:24:58.457 INFO:teuthology.orchestra.run.smithi177.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-22T06:24:58.482 INFO:teuthology.orchestra.run.smithi105.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-22T06:24:58.482 INFO:teuthology.orchestra.run.smithi177.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-22T06:24:58.506 INFO:teuthology.orchestra.run.smithi105.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-22T06:24:58.508 INFO:teuthology.orchestra.run.smithi177.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-22T06:24:58.533 INFO:teuthology.orchestra.run.smithi177.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T06:24:58.558 INFO:teuthology.orchestra.run.smithi177.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-22T06:24:58.650 INFO:teuthology.orchestra.run.smithi177.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-22T06:24:58.675 INFO:teuthology.orchestra.run.smithi177.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 42 kB 00:00 2024-04-22T06:24:58.701 INFO:teuthology.orchestra.run.smithi177.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-04-22T06:24:58.714 INFO:teuthology.orchestra.run.smithi105.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-22T06:24:58.726 INFO:teuthology.orchestra.run.smithi177.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-22T06:24:58.751 INFO:teuthology.orchestra.run.smithi177.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-04-22T06:24:58.777 INFO:teuthology.orchestra.run.smithi177.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-22T06:24:58.802 INFO:teuthology.orchestra.run.smithi177.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-22T06:24:58.827 INFO:teuthology.orchestra.run.smithi177.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-22T06:24:58.852 INFO:teuthology.orchestra.run.smithi177.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-22T06:24:58.878 INFO:teuthology.orchestra.run.smithi177.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-22T06:24:58.881 INFO:teuthology.orchestra.run.smithi105.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-22T06:24:58.903 INFO:teuthology.orchestra.run.smithi177.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-04-22T06:24:58.907 INFO:teuthology.orchestra.run.smithi105.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 4.6 MB/s | 911 kB 00:00 2024-04-22T06:24:58.940 INFO:teuthology.orchestra.run.smithi105.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-22T06:24:58.953 INFO:teuthology.orchestra.run.smithi177.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-04-22T06:24:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:(71/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.0 MB/s | 4.1 MB 00:00 2024-04-22T06:24:59.066 INFO:teuthology.orchestra.run.smithi105.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-22T06:24:59.091 INFO:teuthology.orchestra.run.smithi105.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 952 kB/s | 87 kB 00:00 2024-04-22T06:24:59.150 INFO:teuthology.orchestra.run.smithi105.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-22T06:24:59.154 INFO:teuthology.orchestra.run.smithi177.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-22T06:24:59.179 INFO:teuthology.orchestra.run.smithi177.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-04-22T06:24:59.217 INFO:teuthology.orchestra.run.smithi105.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-04-22T06:24:59.275 INFO:teuthology.orchestra.run.smithi105.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 441 kB/s | 26 kB 00:00 2024-04-22T06:24:59.321 INFO:teuthology.orchestra.run.smithi177.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-04-22T06:24:59.334 INFO:teuthology.orchestra.run.smithi105.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-22T06:24:59.401 INFO:teuthology.orchestra.run.smithi105.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-22T06:24:59.413 INFO:teuthology.orchestra.run.smithi177.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-04-22T06:24:59.438 INFO:teuthology.orchestra.run.smithi177.stdout:(107/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-22T06:24:59.468 INFO:teuthology.orchestra.run.smithi105.stdout:(79/150): postgresql-server-13.14-1.el9.x86_64. 10 MB/s | 5.8 MB 00:00 2024-04-22T06:24:59.471 INFO:teuthology.orchestra.run.smithi177.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-04-22T06:24:59.493 INFO:teuthology.orchestra.run.smithi105.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 35 kB 00:00 2024-04-22T06:24:59.497 INFO:teuthology.orchestra.run.smithi177.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 828 kB/s | 49 kB 00:00 2024-04-22T06:24:59.531 INFO:teuthology.orchestra.run.smithi177.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-22T06:24:59.564 INFO:teuthology.orchestra.run.smithi177.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-22T06:24:59.568 INFO:teuthology.orchestra.run.smithi105.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-22T06:24:59.594 INFO:teuthology.orchestra.run.smithi105.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-22T06:24:59.627 INFO:teuthology.orchestra.run.smithi105.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-22T06:24:59.652 INFO:teuthology.orchestra.run.smithi105.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-22T06:24:59.681 INFO:teuthology.orchestra.run.smithi177.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-22T06:24:59.715 INFO:teuthology.orchestra.run.smithi177.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 838 kB 00:00 2024-04-22T06:24:59.728 INFO:teuthology.orchestra.run.smithi105.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 421 kB/s | 42 kB 00:00 2024-04-22T06:24:59.740 INFO:teuthology.orchestra.run.smithi177.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 740 kB/s | 43 kB 00:00 2024-04-22T06:24:59.761 INFO:teuthology.orchestra.run.smithi105.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-04-22T06:24:59.787 INFO:teuthology.orchestra.run.smithi105.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-22T06:24:59.807 INFO:teuthology.orchestra.run.smithi177.stdout:(115/150): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-22T06:24:59.824 INFO:teuthology.orchestra.run.smithi177.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-22T06:24:59.829 INFO:teuthology.orchestra.run.smithi105.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-22T06:24:59.841 INFO:teuthology.orchestra.run.smithi177.stdout:(117/150): python3-cachetools-4.2.4-1.el9.noarc 256 kB/s | 32 kB 00:00 2024-04-22T06:24:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-04-22T06:24:59.866 INFO:teuthology.orchestra.run.smithi177.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-22T06:24:59.871 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-22T06:24:59.891 INFO:teuthology.orchestra.run.smithi177.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-22T06:24:59.917 INFO:teuthology.orchestra.run.smithi177.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-22T06:24:59.921 INFO:teuthology.orchestra.run.smithi105.stdout:(90/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-22T06:24:59.942 INFO:teuthology.orchestra.run.smithi177.stdout:(121/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-22T06:24:59.946 INFO:teuthology.orchestra.run.smithi105.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 459 kB/s | 54 kB 00:00 2024-04-22T06:24:59.976 INFO:teuthology.orchestra.run.smithi177.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-04-22T06:24:59.979 INFO:teuthology.orchestra.run.smithi105.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 713 kB/s | 42 kB 00:00 2024-04-22T06:25:00.001 INFO:teuthology.orchestra.run.smithi177.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-22T06:25:00.005 INFO:teuthology.orchestra.run.smithi105.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 753 kB/s | 44 kB 00:00 2024-04-22T06:25:00.039 INFO:teuthology.orchestra.run.smithi105.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T06:25:00.043 INFO:teuthology.orchestra.run.smithi177.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-22T06:25:00.064 INFO:teuthology.orchestra.run.smithi105.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 612 kB/s | 36 kB 00:00 2024-04-22T06:25:00.068 INFO:teuthology.orchestra.run.smithi177.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-04-22T06:25:00.097 INFO:teuthology.orchestra.run.smithi105.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-04-22T06:25:00.110 INFO:teuthology.orchestra.run.smithi177.stdout:(126/150): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-22T06:25:00.123 INFO:teuthology.orchestra.run.smithi105.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-22T06:25:00.135 INFO:teuthology.orchestra.run.smithi177.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-22T06:25:00.156 INFO:teuthology.orchestra.run.smithi105.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-04-22T06:25:00.161 INFO:teuthology.orchestra.run.smithi177.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-22T06:25:00.186 INFO:teuthology.orchestra.run.smithi177.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-22T06:25:00.190 INFO:teuthology.orchestra.run.smithi105.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-22T06:25:00.228 INFO:teuthology.orchestra.run.smithi177.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 695 kB/s | 46 kB 00:00 2024-04-22T06:25:00.253 INFO:teuthology.orchestra.run.smithi177.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-22T06:25:00.287 INFO:teuthology.orchestra.run.smithi177.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2024-04-22T06:25:00.329 INFO:teuthology.orchestra.run.smithi177.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-22T06:25:00.354 INFO:teuthology.orchestra.run.smithi177.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-22T06:25:00.388 INFO:teuthology.orchestra.run.smithi177.stdout:(135/150): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-04-22T06:25:00.404 INFO:teuthology.orchestra.run.smithi177.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-22T06:25:00.430 INFO:teuthology.orchestra.run.smithi177.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-22T06:25:00.455 INFO:teuthology.orchestra.run.smithi177.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 458 kB/s | 31 kB 00:00 2024-04-22T06:25:00.480 INFO:teuthology.orchestra.run.smithi177.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-22T06:25:00.506 INFO:teuthology.orchestra.run.smithi177.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-22T06:25:00.531 INFO:teuthology.orchestra.run.smithi177.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-22T06:25:00.556 INFO:teuthology.orchestra.run.smithi177.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-22T06:25:00.582 INFO:teuthology.orchestra.run.smithi177.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-22T06:25:00.582 INFO:teuthology.orchestra.run.smithi105.stdout:(100/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-22T06:25:00.607 INFO:teuthology.orchestra.run.smithi177.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-22T06:25:00.632 INFO:teuthology.orchestra.run.smithi177.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-22T06:25:00.641 INFO:teuthology.orchestra.run.smithi105.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 384 kB/s | 22 kB 00:00 2024-04-22T06:25:00.657 INFO:teuthology.orchestra.run.smithi177.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-22T06:25:00.683 INFO:teuthology.orchestra.run.smithi177.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-22T06:25:00.707 INFO:teuthology.orchestra.run.smithi105.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-22T06:25:00.758 INFO:teuthology.orchestra.run.smithi177.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-22T06:25:00.858 INFO:teuthology.orchestra.run.smithi105.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 3.6 MB/s | 548 kB 00:00 2024-04-22T06:25:00.900 INFO:teuthology.orchestra.run.smithi105.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-04-22T06:25:00.942 INFO:teuthology.orchestra.run.smithi105.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-04-22T06:25:01.073 INFO:teuthology.orchestra.run.smithi177.stdout:(149/150): librados2-19.0.0-3156.g86f7587a.el9. 8.2 MB/s | 3.4 MB 00:00 2024-04-22T06:25:01.084 INFO:teuthology.orchestra.run.smithi105.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-04-22T06:25:01.126 INFO:teuthology.orchestra.run.smithi105.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-04-22T06:25:01.168 INFO:teuthology.orchestra.run.smithi105.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T06:25:01.276 INFO:teuthology.orchestra.run.smithi105.stdout:(109/150): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-04-22T06:25:01.318 INFO:teuthology.orchestra.run.smithi105.stdout:(110/150): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-04-22T06:25:01.342 INFO:teuthology.orchestra.run.smithi177.stdout:(150/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-04-22T06:25:01.344 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:25:01.345 INFO:teuthology.orchestra.run.smithi177.stdout:Total 15 MB/s | 218 MB 00:14 2024-04-22T06:25:01.368 INFO:teuthology.orchestra.run.smithi105.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-04-22T06:25:01.411 INFO:teuthology.orchestra.run.smithi105.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-04-22T06:25:01.444 INFO:teuthology.orchestra.run.smithi105.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-22T06:25:01.486 INFO:teuthology.orchestra.run.smithi105.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 773 kB/s | 32 kB 00:00 2024-04-22T06:25:01.528 INFO:teuthology.orchestra.run.smithi105.stdout:(115/150): python3-certifi-2021.10.8-2.el9.noar 361 kB/s | 15 kB 00:00 2024-04-22T06:25:01.570 INFO:teuthology.orchestra.run.smithi105.stdout:(116/150): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-04-22T06:25:01.612 INFO:teuthology.orchestra.run.smithi105.stdout:(117/150): python3-cherrypy-18.6.1-2.el9.noarch 8.4 MB/s | 358 kB 00:00 2024-04-22T06:25:01.678 INFO:teuthology.orchestra.run.smithi105.stdout:(118/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-04-22T06:25:01.720 INFO:teuthology.orchestra.run.smithi105.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 847 kB/s | 35 kB 00:00 2024-04-22T06:25:01.766 INFO:teuthology.orchestra.run.smithi105.stdout:(120/150): python3-google-auth-2.28.2-1.el9.noa 4.7 MB/s | 203 kB 00:00 2024-04-22T06:25:01.829 INFO:teuthology.orchestra.run.smithi105.stdout:(121/150): python3-grpcio-1.46.7-10.el9.x86_64. 31 MB/s | 2.0 MB 00:00 2024-04-22T06:25:01.871 INFO:teuthology.orchestra.run.smithi105.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-22T06:25:01.913 INFO:teuthology.orchestra.run.smithi105.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-04-22T06:25:01.955 INFO:teuthology.orchestra.run.smithi105.stdout:(124/150): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-04-22T06:25:01.997 INFO:teuthology.orchestra.run.smithi105.stdout:(125/150): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-04-22T06:25:02.039 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:25:02.039 INFO:teuthology.orchestra.run.smithi105.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-04-22T06:25:02.039 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-22T06:25:02.050 INFO:teuthology.orchestra.run.smithi112.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-22T06:25:02.051 INFO:teuthology.orchestra.run.smithi112.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-22T06:25:02.052 INFO:teuthology.orchestra.run.smithi112.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-22T06:25:02.053 INFO:teuthology.orchestra.run.smithi112.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-22T06:25:02.054 INFO:teuthology.orchestra.run.smithi112.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-22T06:25:02.057 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-22T06:25:02.057 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-22T06:25:02.081 INFO:teuthology.orchestra.run.smithi105.stdout:(127/150): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-04-22T06:25:02.118 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-04-22T06:25:02.131 INFO:teuthology.orchestra.run.smithi105.stdout:(128/150): python3-kubernetes-26.1.0-2.el9.noar 20 MB/s | 1.0 MB 00:00 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 6.6 k 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 890 k 2024-04-22T06:25:02.133 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 145 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 1.5 M 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 142 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 3.5 M 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 7.4 M 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 49 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 11 M 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 49 M 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 273 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 763 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 225 k 2024-04-22T06:25:02.134 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 27 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 18 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 32 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 747 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 127 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-22T06:25:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 162 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 320 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 298 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 100 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 85 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 171 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.4 M 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 3.1 M 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-04-22T06:25:02.136 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 21 M 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 34 k 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 2.4 M 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 249 k 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 4.7 M 2024-04-22T06:25:02.137 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 17 M 2024-04-22T06:25:02.138 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.g86f7587a.el9 ceph-noarch 16 k 2024-04-22T06:25:02.138 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 25 k 2024-04-22T06:25:02.138 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:25:02.138 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:25:02.138 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 164 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 500 k 2024-04-22T06:25:02.139 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 5.4 M 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:25:02.140 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 45 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-3156.g86f7587a.el9 ceph 135 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-22T06:25:02.141 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:25:02.142 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:25:02.143 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:25:02.144 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-22T06:25:02.145 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout:Install 148 Packages 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:25:02.146 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 218 M 2024-04-22T06:25:02.147 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-04-22T06:25:02.174 INFO:teuthology.orchestra.run.smithi105.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-22T06:25:02.215 INFO:teuthology.orchestra.run.smithi105.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-22T06:25:02.257 INFO:teuthology.orchestra.run.smithi105.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-04-22T06:25:02.307 INFO:teuthology.orchestra.run.smithi105.stdout:(132/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-22T06:25:02.324 INFO:teuthology.orchestra.run.smithi105.stdout:(133/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 21 kB/s | 45 kB 00:02 2024-04-22T06:25:02.341 INFO:teuthology.orchestra.run.smithi105.stdout:(134/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 29 kB/s | 64 kB 00:02 2024-04-22T06:25:02.367 INFO:teuthology.orchestra.run.smithi105.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 279 kB/s | 16 kB 00:00 2024-04-22T06:25:02.373 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-04-22T06:25:02.383 INFO:teuthology.orchestra.run.smithi105.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 675 kB/s | 28 kB 00:00 2024-04-22T06:25:02.425 INFO:teuthology.orchestra.run.smithi105.stdout:(137/150): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-04-22T06:25:02.450 INFO:teuthology.orchestra.run.smithi105.stdout:(138/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 715 kB/s | 90 kB 00:00 2024-04-22T06:25:02.467 INFO:teuthology.orchestra.run.smithi105.stdout:(139/150): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-22T06:25:02.484 INFO:teuthology.orchestra.run.smithi105.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-22T06:25:02.499 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-04-22T06:25:02.499 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-04-22T06:25:02.501 INFO:teuthology.orchestra.run.smithi105.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 1.1 MB/s | 36 kB 00:00 2024-04-22T06:25:02.526 INFO:teuthology.orchestra.run.smithi105.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 5.3 MB/s | 230 kB 00:00 2024-04-22T06:25:02.552 INFO:teuthology.orchestra.run.smithi105.stdout:(143/150): python3-typing-extensions-4.4.0-2.el 607 kB/s | 51 kB 00:00 2024-04-22T06:25:02.577 INFO:teuthology.orchestra.run.smithi105.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-22T06:25:02.602 INFO:teuthology.orchestra.run.smithi105.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-04-22T06:25:02.628 INFO:teuthology.orchestra.run.smithi105.stdout:(146/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-22T06:25:02.653 INFO:teuthology.orchestra.run.smithi105.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-22T06:25:02.745 INFO:teuthology.orchestra.run.smithi105.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-22T06:25:02.987 INFO:teuthology.orchestra.run.smithi105.stdout:(149/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 9.4 MB/s | 3.1 MB 00:00 2024-04-22T06:25:03.164 INFO:teuthology.orchestra.run.smithi112.stdout:(1/150): ceph-19.0.0-3156.g86f7587a.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-04-22T06:25:03.481 INFO:teuthology.orchestra.run.smithi112.stdout:(2/150): ceph-fuse-19.0.0-3156.g86f7587a.el9.x8 2.7 MB/s | 890 kB 00:00 2024-04-22T06:25:03.573 INFO:teuthology.orchestra.run.smithi112.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.5 MB/s | 145 kB 00:00 2024-04-22T06:25:03.840 INFO:teuthology.orchestra.run.smithi112.stdout:(4/150): ceph-base-19.0.0-3156.g86f7587a.el9.x8 6.1 MB/s | 5.4 MB 00:00 2024-04-22T06:25:03.862 INFO:teuthology.orchestra.run.smithi105.stdout:(150/150): librados2-19.0.0-3156.g86f7587a.el9. 2.8 MB/s | 3.4 MB 00:01 2024-04-22T06:25:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:25:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:Total 13 MB/s | 218 MB 00:16 2024-04-22T06:25:03.898 INFO:teuthology.orchestra.run.smithi112.stdout:(5/150): ceph-mds-19.0.0-3156.g86f7587a.el9.x86 7.3 MB/s | 2.4 MB 00:00 2024-04-22T06:25:04.223 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-04-22T06:25:04.224 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-04-22T06:25:04.374 INFO:teuthology.orchestra.run.smithi112.stdout:(6/150): ceph-mgr-19.0.0-3156.g86f7587a.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-04-22T06:25:04.440 INFO:teuthology.orchestra.run.smithi112.stdout:(7/150): ceph-mon-19.0.0-3156.g86f7587a.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-04-22T06:25:04.910 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-22T06:25:05.035 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-22T06:25:05.035 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-22T06:25:05.074 INFO:teuthology.orchestra.run.smithi112.stdout:(8/150): ceph-common-19.0.0-3156.g86f7587a.el9. 10 MB/s | 21 MB 00:02 2024-04-22T06:25:05.141 INFO:teuthology.orchestra.run.smithi112.stdout:(9/150): ceph-selinux-19.0.0-3156.g86f7587a.el9 378 kB/s | 25 kB 00:00 2024-04-22T06:25:05.666 INFO:teuthology.orchestra.run.smithi112.stdout:(10/150): ceph-radosgw-19.0.0-3156.g86f7587a.el 8.6 MB/s | 11 MB 00:01 2024-04-22T06:25:05.758 INFO:teuthology.orchestra.run.smithi112.stdout:(11/150): cephfs-mirror-19.0.0-3156.g86f7587a.e 2.4 MB/s | 225 kB 00:00 2024-04-22T06:25:05.833 INFO:teuthology.orchestra.run.smithi112.stdout:(12/150): cephfs-shell-19.0.0-3156.g86f7587a.el 361 kB/s | 27 kB 00:00 2024-04-22T06:25:05.917 INFO:teuthology.orchestra.run.smithi112.stdout:(13/150): libcephfs-devel-19.0.0-3156.g86f7587a 380 kB/s | 32 kB 00:00 2024-04-22T06:25:06.084 INFO:teuthology.orchestra.run.smithi112.stdout:(14/150): libcephfs2-19.0.0-3156.g86f7587a.el9. 4.4 MB/s | 747 kB 00:00 2024-04-22T06:25:06.292 INFO:teuthology.orchestra.run.smithi112.stdout:(15/150): libcephsqlite-19.0.0-3156.g86f7587a.e 784 kB/s | 164 kB 00:00 2024-04-22T06:25:06.393 INFO:teuthology.orchestra.run.smithi112.stdout:(16/150): librados-devel-19.0.0-3156.g86f7587a. 1.2 MB/s | 127 kB 00:00 2024-04-22T06:25:06.493 INFO:teuthology.orchestra.run.smithi112.stdout:(17/150): libradosstriper1-19.0.0-3156.g86f7587 4.9 MB/s | 500 kB 00:00 2024-04-22T06:25:06.734 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-22T06:25:06.734 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-22T06:25:06.994 INFO:teuthology.orchestra.run.smithi112.stdout:(18/150): ceph-osd-19.0.0-3156.g86f7587a.el9.x8 6.4 MB/s | 17 MB 00:02 2024-04-22T06:25:07.069 INFO:teuthology.orchestra.run.smithi112.stdout:(19/150): python3-ceph-argparse-19.0.0-3156.g86 604 kB/s | 45 kB 00:00 2024-04-22T06:25:07.153 INFO:teuthology.orchestra.run.smithi112.stdout:(20/150): librgw2-19.0.0-3156.g86f7587a.el9.x86 8.2 MB/s | 5.4 MB 00:00 2024-04-22T06:25:07.178 INFO:teuthology.orchestra.run.smithi112.stdout:(21/150): python3-ceph-common-19.0.0-3156.g86f7 1.2 MB/s | 135 kB 00:00 2024-04-22T06:25:07.228 INFO:teuthology.orchestra.run.smithi112.stdout:(22/150): python3-cephfs-19.0.0-3156.g86f7587a. 2.1 MB/s | 162 kB 00:00 2024-04-22T06:25:07.262 INFO:teuthology.orchestra.run.smithi112.stdout:(23/150): python3-rados-19.0.0-3156.g86f7587a.e 3.7 MB/s | 320 kB 00:00 2024-04-22T06:25:07.312 INFO:teuthology.orchestra.run.smithi112.stdout:(24/150): python3-rbd-19.0.0-3156.g86f7587a.el9 3.5 MB/s | 298 kB 00:00 2024-04-22T06:25:07.346 INFO:teuthology.orchestra.run.smithi112.stdout:(25/150): python3-rgw-19.0.0-3156.g86f7587a.el9 1.2 MB/s | 100 kB 00:00 2024-04-22T06:25:07.388 INFO:teuthology.orchestra.run.smithi112.stdout:(26/150): rbd-fuse-19.0.0-3156.g86f7587a.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-22T06:25:07.479 INFO:teuthology.orchestra.run.smithi112.stdout:(27/150): rbd-nbd-19.0.0-3156.g86f7587a.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-22T06:25:07.572 INFO:teuthology.orchestra.run.smithi112.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3156.g 370 kB/s | 34 kB 00:00 2024-04-22T06:25:07.672 INFO:teuthology.orchestra.run.smithi112.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3156.g86f7587 1.4 MB/s | 142 kB 00:00 2024-04-22T06:25:07.856 INFO:teuthology.orchestra.run.smithi112.stdout:(30/150): rbd-mirror-19.0.0-3156.g86f7587a.el9. 6.1 MB/s | 3.1 MB 00:00 2024-04-22T06:25:08.373 INFO:teuthology.orchestra.run.smithi112.stdout:(31/150): ceph-test-19.0.0-3156.g86f7587a.el9.x 15 MB/s | 49 MB 00:03 2024-04-22T06:25:08.432 INFO:teuthology.orchestra.run.smithi112.stdout:(32/150): ceph-mgr-dashboard-19.0.0-3156.g86f75 4.7 MB/s | 3.5 MB 00:00 2024-04-22T06:25:08.457 INFO:teuthology.orchestra.run.smithi112.stdout:(33/150): ceph-mgr-modules-core-19.0.0-3156.g86 2.9 MB/s | 249 kB 00:00 2024-04-22T06:25:08.499 INFO:teuthology.orchestra.run.smithi112.stdout:(34/150): ceph-mgr-rook-19.0.0-3156.g86f7587a.e 740 kB/s | 49 kB 00:00 2024-04-22T06:25:08.524 INFO:teuthology.orchestra.run.smithi112.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3156.g8 246 kB/s | 16 kB 00:00 2024-04-22T06:25:08.566 INFO:teuthology.orchestra.run.smithi112.stdout:(36/150): ceph-volume-19.0.0-3156.g86f7587a.el9 4.0 MB/s | 273 kB 00:00 2024-04-22T06:25:08.633 INFO:teuthology.orchestra.run.smithi112.stdout:(37/150): cephfs-top-19.0.0-3156.g86f7587a.el9. 274 kB/s | 18 kB 00:00 2024-04-22T06:25:08.658 INFO:teuthology.orchestra.run.smithi112.stdout:(38/150): cephadm-19.0.0-3156.g86f7587a.el9.noa 5.6 MB/s | 763 kB 00:00 2024-04-22T06:25:08.825 INFO:teuthology.orchestra.run.smithi112.stdout:(39/150): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-04-22T06:25:09.400 INFO:teuthology.orchestra.run.smithi112.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 105 kB/s | 80 kB 00:00 2024-04-22T06:25:09.709 INFO:teuthology.orchestra.run.smithi112.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 221 kB/s | 195 kB 00:00 2024-04-22T06:25:10.259 INFO:teuthology.orchestra.run.smithi112.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 502 kB/s | 803 kB 00:01 2024-04-22T06:25:10.318 INFO:teuthology.orchestra.run.smithi112.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 276 kB/s | 253 kB 00:00 2024-04-22T06:25:10.509 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-04-22T06:25:10.585 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:25:10.611 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:25:10.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:25:10.982 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:25:11.001 INFO:teuthology.orchestra.run.smithi112.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 144 kB/s | 106 kB 00:00 2024-04-22T06:25:11.068 INFO:teuthology.orchestra.run.smithi112.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 180 kB/s | 135 kB 00:00 2024-04-22T06:25:11.095 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:11.217 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:11.237 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:11.331 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:11.373 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:25:11.402 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:25:11.428 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:25:11.581 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:25:11.642 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:25:11.670 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:11.760 INFO:teuthology.orchestra.run.smithi112.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 165 kB/s | 125 kB 00:00 2024-04-22T06:25:11.773 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:11.796 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:11.835 INFO:teuthology.orchestra.run.smithi112.stdout:(47/150): python3-cryptography-36.0.1-4.el9.x86 590 kB/s | 1.2 MB 00:02 2024-04-22T06:25:11.858 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:11.880 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:11.903 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:11.919 INFO:teuthology.orchestra.run.smithi112.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 252 kB/s | 215 kB 00:00 2024-04-22T06:25:12.080 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:12.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:12.243 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:25:12.336 INFO:teuthology.orchestra.run.smithi112.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 257 kB/s | 107 kB 00:00 2024-04-22T06:25:12.347 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:25:12.380 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:25:12.426 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:25:12.436 INFO:teuthology.orchestra.run.smithi112.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-04-22T06:25:12.503 INFO:teuthology.orchestra.run.smithi112.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 928 kB/s | 689 kB 00:00 2024-04-22T06:25:12.505 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:25:12.552 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-22T06:25:12.564 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:25:12.595 INFO:teuthology.orchestra.run.smithi112.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-04-22T06:25:12.607 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:25:12.628 INFO:teuthology.orchestra.run.smithi112.stdout:(53/150): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-22T06:25:12.640 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:25:12.668 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:25:12.704 INFO:teuthology.orchestra.run.smithi112.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-22T06:25:12.715 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:25:12.721 INFO:teuthology.orchestra.run.smithi112.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 198 kB/s | 18 kB 00:00 2024-04-22T06:25:12.738 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:25:12.796 INFO:teuthology.orchestra.run.smithi112.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-22T06:25:12.799 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:25:12.829 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:25:12.830 INFO:teuthology.orchestra.run.smithi112.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-22T06:25:12.840 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:25:12.873 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:25:12.874 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:25:12.897 INFO:teuthology.orchestra.run.smithi112.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-04-22T06:25:12.947 INFO:teuthology.orchestra.run.smithi112.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-22T06:25:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:25:12.984 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:25:12.997 INFO:teuthology.orchestra.run.smithi112.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-04-22T06:25:13.030 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:25:13.056 INFO:teuthology.orchestra.run.smithi112.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 235 kB/s | 25 kB 00:00 2024-04-22T06:25:13.056 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:25:13.098 INFO:teuthology.orchestra.run.smithi112.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-04-22T06:25:13.128 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:25:13.164 INFO:teuthology.orchestra.run.smithi112.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-22T06:25:13.180 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:25:13.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:25:13.206 INFO:teuthology.orchestra.run.smithi112.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-22T06:25:13.240 INFO:teuthology.orchestra.run.smithi112.stdout:(65/150): libtool-2.4.6-45.el9.x86_64.rpm 1.7 MB/s | 582 kB 00:00 2024-04-22T06:25:13.256 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:25:13.265 INFO:teuthology.orchestra.run.smithi112.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 35 kB 00:00 2024-04-22T06:25:13.316 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:13.390 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:25:13.439 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:13.440 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:25:13.467 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:13.516 INFO:teuthology.orchestra.run.smithi112.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 6.4 MB/s | 1.6 MB 00:00 2024-04-22T06:25:13.542 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:13.583 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:25:13.613 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:25:13.638 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:25:13.675 INFO:teuthology.orchestra.run.smithi112.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 5.6 MB/s | 911 kB 00:00 2024-04-22T06:25:13.795 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:25:13.816 INFO:teuthology.orchestra.run.smithi112.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 942 kB/s | 133 kB 00:00 2024-04-22T06:25:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:25:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:13.900 INFO:teuthology.orchestra.run.smithi112.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 4.1 MB 00:00 2024-04-22T06:25:13.982 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:25:13.992 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:14.024 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:14.070 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:25:14.086 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:14.098 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:25:14.116 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:14.138 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:14.141 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:25:14.179 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:25:14.212 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:25:14.268 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:25:14.333 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:14.340 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:25:14.365 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:14.371 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:25:14.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:25:14.446 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:25:14.480 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:25:14.482 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:25:14.526 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:25:14.568 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:25:14.571 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:25:14.606 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:25:14.641 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:25:14.705 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:25:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:25:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:25:14.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:25:14.779 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:25:14.800 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:25:14.820 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:25:14.841 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:25:14.870 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:25:14.915 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:25:14.939 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:25:14.992 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:25:15.020 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:25:15.021 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:25:15.057 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:25:15.060 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:25:15.086 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:25:15.110 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:25:15.135 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:25:15.168 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:25:15.185 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:25:15.211 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:25:15.240 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:25:15.313 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:25:15.314 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:25:15.316 INFO:teuthology.orchestra.run.smithi112.stdout:(71/150): protobuf-3.14.0-13.el9.x86_64.rpm 729 kB/s | 1.0 MB 00:01 2024-04-22T06:25:15.351 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:25:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:25:15.379 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:25:15.383 INFO:teuthology.orchestra.run.smithi112.stdout:(72/150): openblas-openmp-0.3.21-2.el9.x86_64.r 2.2 MB/s | 4.7 MB 00:02 2024-04-22T06:25:15.414 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:25:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:25:15.458 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:25:15.509 INFO:teuthology.orchestra.run.smithi112.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 456 kB/s | 87 kB 00:00 2024-04-22T06:25:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:25:15.552 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:25:15.579 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:25:15.582 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:25:15.608 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:25:15.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:25:15.633 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:25:15.675 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:25:15.709 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:25:15.800 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:25:15.848 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:25:15.859 INFO:teuthology.orchestra.run.smithi112.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 698 kB/s | 244 kB 00:00 2024-04-22T06:25:16.097 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:25:16.147 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:25:16.176 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:25:16.218 INFO:teuthology.orchestra.run.smithi112.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 691 kB/s | 248 kB 00:00 2024-04-22T06:25:16.254 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:25:16.258 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:25:16.282 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:25:16.318 INFO:teuthology.orchestra.run.smithi112.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-04-22T06:25:16.323 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:25:16.324 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:25:16.354 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:25:16.387 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:25:16.426 INFO:teuthology.orchestra.run.smithi112.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 177 kB/s | 19 kB 00:00 2024-04-22T06:25:16.443 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:25:16.517 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:25:16.555 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:25:16.591 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:25:16.639 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:25:16.674 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:25:16.685 INFO:teuthology.orchestra.run.smithi112.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 666 kB/s | 172 kB 00:00 2024-04-22T06:25:16.718 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:25:16.760 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:25:16.802 INFO:teuthology.orchestra.run.smithi112.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 298 kB/s | 35 kB 00:00 2024-04-22T06:25:16.908 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:25:16.956 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:25:16.986 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:25:17.047 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:25:17.055 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:25:17.103 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:25:17.137 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:25:17.158 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:25:17.203 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:25:17.238 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:25:17.243 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:25:17.268 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:25:17.293 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:25:17.312 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:25:17.371 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:25:17.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:25:17.448 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:25:17.496 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:25:17.542 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:25:17.570 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:25:17.604 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:25:17.625 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:25:17.649 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:25:17.653 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:25:17.686 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:25:17.726 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:25:17.731 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:25:17.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:25:17.790 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:25:17.824 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:25:17.866 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:25:17.891 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:25:17.981 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:25:18.030 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:25:18.061 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:25:18.109 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:25:18.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:25:18.395 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:25:18.497 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:25:18.528 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:25:19.253 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:25:19.302 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:25:19.302 INFO:teuthology.orchestra.run.smithi112.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 1.1 MB/s | 5.8 MB 00:05 2024-04-22T06:25:19.336 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:25:19.440 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:25:19.495 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:25:19.587 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:25:19.638 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:25:19.652 INFO:teuthology.orchestra.run.smithi112.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 716 kB/s | 250 kB 00:00 2024-04-22T06:25:19.776 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:25:19.830 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:25:19.858 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:25:19.860 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:25:19.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:25:19.919 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:25:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:25:20.061 INFO:teuthology.orchestra.run.smithi112.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 544 kB/s | 222 kB 00:00 2024-04-22T06:25:20.231 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:25:20.253 INFO:teuthology.orchestra.run.smithi112.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 403 kB/s | 77 kB 00:00 2024-04-22T06:25:20.274 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:25:20.354 INFO:teuthology.orchestra.run.smithi112.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 422 kB/s | 42 kB 00:00 2024-04-22T06:25:20.356 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:25:20.387 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:25:20.423 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:25:20.606 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:25:20.727 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:25:20.814 INFO:teuthology.orchestra.run.smithi112.stdout:(85/150): python3-protobuf-3.14.0-13.el9.noarch 588 kB/s | 269 kB 00:00 2024-04-22T06:25:20.903 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:25:20.927 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:25:20.998 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:25:21.020 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:25:21.079 INFO:teuthology.orchestra.run.smithi112.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 595 kB/s | 159 kB 00:00 2024-04-22T06:25:21.521 INFO:teuthology.orchestra.run.smithi112.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 631 kB/s | 279 kB 00:00 2024-04-22T06:25:21.621 INFO:teuthology.orchestra.run.smithi112.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 510 kB/s | 51 kB 00:00 2024-04-22T06:25:21.721 INFO:teuthology.orchestra.run.smithi112.stdout:(89/150): python3-requests-oauthlib-1.3.0-12.el 536 kB/s | 54 kB 00:00 2024-04-22T06:25:21.991 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:25:22.081 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:25:22.108 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:25:22.142 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:25:22.459 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:25:22.496 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:25:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:25:22.617 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:25:22.653 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:25:23.143 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:25:23.165 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:25:23.237 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:25:23.259 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:25:23.331 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:23.358 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:23.463 INFO:teuthology.orchestra.run.smithi112.stdout:(90/150): python3-babel-2.9.1-2.el9.noarch.rpm 756 kB/s | 6.0 MB 00:08 2024-04-22T06:25:23.613 INFO:teuthology.orchestra.run.smithi112.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 766 kB/s | 5.1 MB 00:06 2024-04-22T06:25:23.630 INFO:teuthology.orchestra.run.smithi112.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 250 kB/s | 42 kB 00:00 2024-04-22T06:25:23.705 INFO:teuthology.orchestra.run.smithi112.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 481 kB/s | 44 kB 00:00 2024-04-22T06:25:23.814 INFO:teuthology.orchestra.run.smithi112.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-04-22T06:25:23.904 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:23.914 INFO:teuthology.orchestra.run.smithi112.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-22T06:25:23.937 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:25:23.999 INFO:teuthology.orchestra.run.smithi112.stdout:(96/150): qatlib-24.02.0-1.el9.x86_64.rpm 600 kB/s | 221 kB 00:00 2024-04-22T06:25:24.031 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:25:24.031 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:25:24.031 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:25:24.032 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:24.063 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:25:24.107 INFO:teuthology.orchestra.run.smithi112.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 534 kB/s | 58 kB 00:00 2024-04-22T06:25:24.291 INFO:teuthology.orchestra.run.smithi112.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 511 kB/s | 94 kB 00:00 2024-04-22T06:25:24.349 INFO:teuthology.orchestra.run.smithi112.stdout:(99/150): socat-1.7.4.1-5.el9.x86_64.rpm 702 kB/s | 305 kB 00:00 2024-04-22T06:25:24.465 INFO:teuthology.orchestra.run.smithi112.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 448 kB/s | 45 kB 00:00 2024-04-22T06:25:24.475 INFO:teuthology.orchestra.run.smithi112.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-22T06:25:24.533 INFO:teuthology.orchestra.run.smithi112.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 288 kB/s | 22 kB 00:00 2024-04-22T06:25:24.594 INFO:teuthology.orchestra.run.smithi112.stdout:(103/150): protobuf-compiler-3.14.0-13.el9.x86_ 7.1 MB/s | 863 kB 00:00 2024-04-22T06:25:25.044 INFO:teuthology.orchestra.run.smithi112.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-04-22T06:25:25.078 INFO:teuthology.orchestra.run.smithi112.stdout:(105/150): abseil-cpp-20211102.0-3.el9.x86_64.r 996 kB/s | 548 kB 00:00 2024-04-22T06:25:25.203 INFO:teuthology.orchestra.run.smithi112.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 122 kB/s | 19 kB 00:00 2024-04-22T06:25:25.320 INFO:teuthology.orchestra.run.smithi112.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-22T06:25:25.420 INFO:teuthology.orchestra.run.smithi112.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-04-22T06:25:25.487 INFO:teuthology.orchestra.run.smithi112.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-22T06:25:25.513 INFO:teuthology.orchestra.run.smithi112.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-22T06:25:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:25.654 INFO:teuthology.orchestra.run.smithi112.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-04-22T06:25:25.696 INFO:teuthology.orchestra.run.smithi112.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-04-22T06:25:25.755 INFO:teuthology.orchestra.run.smithi112.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-22T06:25:25.797 INFO:teuthology.orchestra.run.smithi112.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-04-22T06:25:25.847 INFO:teuthology.orchestra.run.smithi112.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 349 kB/s | 32 kB 00:00 2024-04-22T06:25:25.898 INFO:teuthology.orchestra.run.smithi112.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-22T06:25:25.956 INFO:teuthology.orchestra.run.smithi112.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-22T06:25:26.048 INFO:teuthology.orchestra.run.smithi112.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-22T06:25:26.115 INFO:teuthology.orchestra.run.smithi112.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-22T06:25:26.132 INFO:teuthology.orchestra.run.smithi112.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-22T06:25:26.149 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:25:26.200 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:25:26.215 INFO:teuthology.orchestra.run.smithi112.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-04-22T06:25:26.293 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:25:26.293 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:25:26.294 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:25:26.294 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:26.326 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:25:26.332 INFO:teuthology.orchestra.run.smithi112.stdout:(122/150): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-22T06:25:26.358 INFO:teuthology.orchestra.run.smithi112.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-22T06:25:26.425 INFO:teuthology.orchestra.run.smithi112.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-22T06:25:26.450 INFO:teuthology.orchestra.run.smithi112.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-22T06:25:26.517 INFO:teuthology.orchestra.run.smithi112.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-22T06:25:26.542 INFO:teuthology.orchestra.run.smithi112.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-22T06:25:26.609 INFO:teuthology.orchestra.run.smithi112.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-22T06:25:26.726 INFO:teuthology.orchestra.run.smithi112.stdout:(129/150): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-04-22T06:25:26.827 INFO:teuthology.orchestra.run.smithi112.stdout:(130/150): python3-more-itertools-8.12.0-2.el9. 786 kB/s | 79 kB 00:00 2024-04-22T06:25:26.919 INFO:teuthology.orchestra.run.smithi112.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-04-22T06:25:27.086 INFO:teuthology.orchestra.run.smithi112.stdout:(132/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-22T06:25:27.211 INFO:teuthology.orchestra.run.smithi112.stdout:(133/150): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-04-22T06:25:27.278 INFO:teuthology.orchestra.run.smithi112.stdout:(134/150): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-04-22T06:25:27.387 INFO:teuthology.orchestra.run.smithi112.stdout:(135/150): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-22T06:25:27.412 INFO:teuthology.orchestra.run.smithi112.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 448 kB/s | 90 kB 00:00 2024-04-22T06:25:27.479 INFO:teuthology.orchestra.run.smithi112.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-04-22T06:25:27.604 INFO:teuthology.orchestra.run.smithi112.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 982 kB/s | 188 kB 00:00 2024-04-22T06:25:27.630 INFO:teuthology.orchestra.run.smithi112.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-04-22T06:25:27.696 INFO:teuthology.orchestra.run.smithi112.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-22T06:25:27.721 INFO:teuthology.orchestra.run.smithi112.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 555 kB/s | 51 kB 00:00 2024-04-22T06:25:27.830 INFO:teuthology.orchestra.run.smithi112.stdout:(142/150): python3-websocket-client-1.2.3-2.el9 827 kB/s | 90 kB 00:00 2024-04-22T06:25:27.864 INFO:teuthology.orchestra.run.smithi112.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-04-22T06:25:27.964 INFO:teuthology.orchestra.run.smithi112.stdout:(144/150): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-22T06:25:27.989 INFO:teuthology.orchestra.run.smithi112.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-04-22T06:25:28.065 INFO:teuthology.orchestra.run.smithi112.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-22T06:25:28.115 INFO:teuthology.orchestra.run.smithi112.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-22T06:25:28.515 INFO:teuthology.orchestra.run.smithi112.stdout:(148/150): librados2-19.0.0-3156.g86f7587a.el9. 7.5 MB/s | 3.4 MB 00:00 2024-04-22T06:25:28.660 INFO:teuthology.orchestra.run.smithi112.stdout:(149/150): librbd1-19.0.0-3156.g86f7587a.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-04-22T06:25:35.217 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:25:35.217 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-04-22T06:25:35.217 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-04-22T06:25:35.218 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:35.392 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:25:35.407 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:25:35.407 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:35.407 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:25:35.407 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:36.921 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:25:36.946 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:25:36.946 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:36.946 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:25:36.947 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:25:36.947 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:25:36.947 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:37.172 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:25:37.194 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:37.631 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:25:37.653 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:37.769 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-04-22T06:25:37.781 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-04-22T06:25:37.782 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:37.826 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:25:37.881 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:25:37.996 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:25:38.011 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:25:38.012 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:38.012 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:25:38.012 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:38.066 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:25:38.114 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:25:38.184 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:25:38.273 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:25:38.412 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:25:38.457 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:25:38.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:25:38.516 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:25:38.926 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:25:38.956 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:25:38.988 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:25:39.049 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:25:39.080 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:25:39.483 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:25:39.508 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:39.733 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:25:39.756 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:40.124 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:25:40.156 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:25:40.201 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:25:40.223 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:25:40.223 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:40.224 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:25:40.224 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:25:40.224 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:25:40.224 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:40.331 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:25:40.388 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:25:40.444 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:25:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:25:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:25:40.754 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:25:40.843 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:25:40.983 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:25:41.018 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:25:41.050 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:25:41.077 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:25:41.125 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:25:41.149 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:25:41.249 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:25:41.412 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:25:41.434 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:25:41.456 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:25:41.457 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:41.457 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:25:41.457 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:25:41.457 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:25:41.457 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:41.462 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:25:41.493 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:25:41.501 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:25:41.512 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:25:41.525 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:25:41.586 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:25:41.610 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:25:41.624 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:25:41.658 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:25:42.583 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:25:42.612 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:25:42.634 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:25:42.635 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:42.635 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:25:42.635 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:25:42.635 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:25:42.635 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:42.657 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:25:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:25:42.701 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:25:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:25:42.701 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:42.702 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:25:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:25:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:43.072 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:25:43.095 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:25:43.095 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:43.095 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:25:43.095 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:25:43.095 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:25:43.096 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:25:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:25:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:25:43.948 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:25:43.986 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:25:44.010 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:44.046 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:25:44.056 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:25:44.131 INFO:teuthology.orchestra.run.smithi112.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 743 kB/s | 16 MB 00:22 2024-04-22T06:25:44.134 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:25:44.135 INFO:teuthology.orchestra.run.smithi112.stdout:Total 5.2 MB/s | 218 MB 00:41 2024-04-22T06:25:44.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:25:44.202 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:25:45.124 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:25:45.130 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-04-22T06:25:45.156 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:25:45.179 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:45.224 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:25:45.246 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:25:45.246 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:45.246 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:25:45.246 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:45.255 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-04-22T06:25:45.255 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-04-22T06:25:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:25:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:45.617 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:25:45.639 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:25:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-04-22T06:25:46.954 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-04-22T06:25:47.709 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:25:47.750 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:25:47.792 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:25:47.823 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:25:47.850 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:25:48.034 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:25:48.088 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:25:48.119 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:25:48.732 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:48.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:48.786 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:49.564 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:25:49.654 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:25:49.695 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:25:49.777 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:25:49.809 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:25:49.893 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:25:49.893 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:25:50.245 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:25:50.287 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:25:50.331 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:25:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:25:50.387 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:25:50.584 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:25:50.624 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:25:50.654 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:25:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:25:52.047 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:25:52.130 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:25:52.171 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:25:52.253 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:25:52.285 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:25:52.285 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:25:52.368 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:25:52.369 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:25:52.756 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-04-22T06:25:52.825 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-22T06:25:52.850 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-22T06:25:52.902 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-22T06:25:53.232 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-22T06:25:53.335 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 5/152 2024-04-22T06:25:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:53.569 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:53.610 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:25:53.640 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-22T06:25:53.666 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-22T06:25:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-22T06:25:53.863 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-22T06:25:53.891 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:53.994 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:54.017 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:54.071 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 13/152 2024-04-22T06:25:54.093 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:54.110 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-22T06:25:54.276 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:54.309 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-22T06:25:54.416 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-22T06:25:54.505 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-22T06:25:54.541 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-22T06:25:54.576 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-22T06:25:54.642 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-22T06:25:54.684 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-22T06:25:54.728 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-22T06:25:54.768 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-22T06:25:54.796 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-22T06:25:54.865 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-22T06:25:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-22T06:25:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-22T06:25:55.010 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-22T06:25:55.086 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-22T06:25:55.120 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 30/152 2024-04-22T06:25:55.155 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 31/152 2024-04-22T06:25:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 32/152 2024-04-22T06:25:55.254 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-22T06:25:55.299 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-22T06:25:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-22T06:25:55.533 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-22T06:25:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-22T06:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:25:55.653 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:55.654 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:25:55.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:25:55.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:25:55.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:25:55.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:25:55.659 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:25:55.660 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:25:55.661 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:25:55.662 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:25:55.663 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:25:55.664 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:25:55.665 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:25:55.666 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:25:56.118 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-22T06:25:56.198 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-22T06:25:56.234 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-22T06:25:56.277 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-22T06:25:56.412 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 42/152 2024-04-22T06:25:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-22T06:25:56.621 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-22T06:25:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-22T06:25:56.724 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-22T06:25:56.761 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-22T06:25:56.808 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-22T06:25:56.846 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-22T06:25:56.887 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-22T06:25:56.929 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-22T06:25:57.067 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-22T06:25:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-22T06:25:57.140 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-22T06:25:57.181 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-22T06:25:57.277 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-22T06:25:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-22T06:25:57.363 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-22T06:25:57.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-22T06:25:57.430 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-22T06:25:57.505 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-22T06:25:57.633 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-22T06:25:57.671 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-22T06:25:57.699 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-22T06:25:57.701 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:25:57.701 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:57.702 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-04-22T06:25:57.702 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.702 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.702 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-04-22T06:25:57.702 INFO:teuthology.orchestra.run.smithi177.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.703 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.704 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.704 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.704 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.704 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.704 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.705 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.706 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:25:57.707 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:25:57.708 INFO:teuthology.orchestra.run.smithi177.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.709 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:25:57.710 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:25:57.710 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:25:57.710 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:25:57.710 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:25:57.710 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:25:57.711 INFO:teuthology.orchestra.run.smithi177.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:25:57.712 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:25:57.713 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:25:57.714 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:25:57.715 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:25:57.716 INFO:teuthology.orchestra.run.smithi177.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:25:57.716 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:25:57.716 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:25:57.716 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:25:57.716 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:25:57.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:25:57.718 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:25:57.719 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:25:57.720 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.721 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:25:57.722 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:25:57.723 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.724 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:25:57.725 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:25:57.726 INFO:teuthology.orchestra.run.smithi177.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:25:57.726 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:25:57.727 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-04-22T06:25:57.733 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-22T06:25:57.777 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-22T06:25:57.815 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-22T06:25:57.846 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-22T06:25:57.876 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-22T06:25:57.902 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-22T06:25:57.935 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-22T06:25:57.977 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-22T06:25:58.011 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-22T06:25:58.080 DEBUG:teuthology.parallel:result is None 2024-04-22T06:25:58.098 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-22T06:25:58.142 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-22T06:25:58.182 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-22T06:25:58.228 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-22T06:25:58.327 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-22T06:25:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-22T06:25:58.427 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:25:58.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:25:58.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:25:58.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:25:58.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:25:58.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:25:58.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:25:58.442 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:25:58.443 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:25:58.444 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:25:58.445 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:25:58.446 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:25:58.447 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:25:58.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:25:58.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:25:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-22T06:25:59.172 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-22T06:25:59.205 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-22T06:25:59.304 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-22T06:25:59.372 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-22T06:25:59.460 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-22T06:25:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-22T06:25:59.668 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-22T06:25:59.697 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-22T06:25:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-22T06:26:00.425 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.537 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.538 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.539 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.540 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:26:00.541 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:26:00.542 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:26:00.543 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:26:00.544 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:26:00.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:26:00.546 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:26:00.547 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:26:00.548 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:26:00.549 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:26:00.549 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:26:00.549 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-22T06:26:00.555 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-22T06:26:00.895 DEBUG:teuthology.parallel:result is None 2024-04-22T06:26:01.801 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-22T06:26:01.902 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-22T06:26:01.930 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-22T06:26:01.963 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-22T06:26:02.275 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-22T06:26:02.317 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 97/152 2024-04-22T06:26:02.455 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 98/152 2024-04-22T06:26:02.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-22T06:26:02.970 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-22T06:26:02.995 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:26:03.066 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 101/152 2024-04-22T06:26:03.080 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 102/152 2024-04-22T06:26:05.270 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:26:05.310 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:26:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 103/152 2024-04-22T06:26:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:26:05.964 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 104/152 2024-04-22T06:26:05.964 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:26:05.964 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:26:05.964 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 105/152 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-04-22T06:26:17.279 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-04-22T06:26:17.280 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-04-22T06:26:17.280 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-04-22T06:26:17.280 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:17.469 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:26:17.485 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 106/152 2024-04-22T06:26:17.485 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:17.485 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:26:17.485 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:18.972 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:26:18.997 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 107/152 2024-04-22T06:26:18.997 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:18.997 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:26:18.998 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:26:18.998 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:26:18.998 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:19.223 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:26:19.245 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 108/152 2024-04-22T06:26:19.245 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:19.245 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:26:19.245 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:26:19.245 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:26:19.246 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:19.690 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:26:19.712 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 109/152 2024-04-22T06:26:19.712 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:19.712 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:26:19.713 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:26:19.713 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:26:19.713 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:19.812 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-22T06:26:19.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-22T06:26:19.933 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-22T06:26:20.117 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-22T06:26:20.166 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-22T06:26:20.243 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-22T06:26:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-22T06:26:20.479 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-22T06:26:20.516 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-22T06:26:20.548 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-22T06:26:20.584 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-22T06:26:20.977 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:26:21.008 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 121/152 2024-04-22T06:26:21.040 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 122/152 2024-04-22T06:26:21.101 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 123/152 2024-04-22T06:26:21.131 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:26:22.164 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 124/152 2024-04-22T06:26:22.208 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:26:23.165 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 125/152 2024-04-22T06:26:23.200 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:26:23.302 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 126/152 2024-04-22T06:26:23.479 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 127/152 2024-04-22T06:26:23.511 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 128/152 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:26:23.533 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:26:23.589 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 129/152 2024-04-22T06:26:23.686 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 130/152 2024-04-22T06:26:23.734 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 131/152 2024-04-22T06:26:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-22T06:26:24.680 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:26:24.702 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 133/152 2024-04-22T06:26:24.703 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:24.703 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:26:24.703 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:26:24.703 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:26:24.703 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:24.756 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:26:24.778 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.g86f75 134/152 2024-04-22T06:26:24.778 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:24.778 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:26:24.778 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:24.832 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 135/152 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-22T06:26:24.854 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:25.166 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:26:25.187 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 136/152 2024-04-22T06:26:25.187 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:26:25.187 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:26:25.188 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:26:25.188 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:26:25.188 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 137/152 2024-04-22T06:26:29.844 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 138/152 2024-04-22T06:26:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 139/152 2024-04-22T06:26:29.917 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-22T06:26:29.944 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-22T06:26:30.124 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 142/152 2024-04-22T06:26:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-22T06:26:30.212 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 144/152 2024-04-22T06:26:30.807 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:26:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:26:30.863 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-22T06:26:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 146/152 2024-04-22T06:26:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-22T06:26:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-22T06:26:31.861 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-22T06:26:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-22T06:26:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:26:32.002 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-22T06:26:32.003 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:26:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:26:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 1/152 2024-04-22T06:26:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 3/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 4/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.g86f75 5/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 6/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 7/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 8/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 9/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 10/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 11/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 12/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 13/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 14/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_ 15/152 2024-04-22T06:26:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 16/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 17/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_6 18/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86 19/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 20/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el 21/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.g86f7587a.el9. 22/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_6 23/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 24/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 25/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 26/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 27/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 28/152 2024-04-22T06:26:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 29/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a. 30/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noa 31/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.n 32/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f 33/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el 34/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 35/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.e 36/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 37/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 38/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 39/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-22T06:26:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-22T06:26:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-22T06:26:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-22T06:26:38.030 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-22T06:26:38.031 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-22T06:26:38.032 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-22T06:26:38.033 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-22T06:26:38.034 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 149/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-22T06:26:38.035 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 151/152 2024-04-22T06:26:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-22T06:26:40.040 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:40.040 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: automake-1.16.2-8.el9.noarch 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: bison-3.7.4-5.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.041 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.042 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top-2:19.0.0-3156.g86f7587a.el9.noarch 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: flex-2.6.4-9.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-22T06:26:40.043 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:26:40.044 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:26:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-13.14-1.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-22T06:26:40.046 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:26:40.047 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:26:40.048 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:26:40.049 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:26:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:26:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:26:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:26:40.051 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:26:40.052 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:26:40.053 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-3156.g86f7587a.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-22T06:26:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:26:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-22T06:26:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-04-22T06:26:40.055 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-04-22T06:26:40.405 DEBUG:teuthology.parallel:result is None 2024-04-22T06:26:40.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:40.566 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:26:40.591 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:40.592 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:40.592 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:26:40.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:40.828 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:26:40.850 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:40.850 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:40.850 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:26:40.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:41.009 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:26:41.035 INFO:teuthology.orchestra.run.smithi177.stdout:19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:41.035 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.g86f7587a.el9 2024-04-22T06:26:41.036 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.g86f7587a is installed. 2024-04-22T06:26:41.037 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:26:41.038 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:26:41.038 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:26:41.069 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:26:41.069 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:26:41.095 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:26:41.095 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:26:41.126 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:26:41.126 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:26:41.126 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:26:41.155 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:26:41.222 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:26:41.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:26:41.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:26:41.310 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:26:41.310 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:26:41.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:26:41.406 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:26:41.406 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:26:41.406 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:26:41.433 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:26:41.497 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:26:41.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:26:41.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:26:41.585 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:26:41.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:26:41.609 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:26:41.674 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:26:41.674 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:26:41.674 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:26:41.699 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:26:41.762 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:26:41.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:26:41.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:26:41.849 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:26:41.849 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:26:41.873 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:26:41.937 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T06:26:42.061 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-22T06:26:42.062 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:42.062 INFO:tasks.cephadm:Cluster fsid is 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:26:42.062 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T06:26:42.063 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.105', 'mon.b': '172.21.15.112', 'mon.c': '172.21.15.177'} 2024-04-22T06:26:42.063 INFO:tasks.cephadm:First mon is mon.a on smithi105 2024-04-22T06:26:42.063 INFO:tasks.cephadm:First mgr is x 2024-04-22T06:26:42.063 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T06:26:42.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-04-22T06:26:42.088 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-04-22T06:26:42.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-04-22T06:26:42.137 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-22T06:26:42.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:42.292 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 22:42:52.445656', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '86f7587a5a09af35f0895e1a2d08527638fae697', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78786/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-g86f7587a', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.g86f7587a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-21 20:57:03.749940', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-22T06:26:42.522 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 86f7587a5a09af35f0895e1a2d08527638fae697 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:26:42.523 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm 2024-04-22T06:26:42.523 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:43.138 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:26 /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:43.138 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:43.563 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:26 /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:43.564 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/86f7587a5a09af35f0895e1a2d08527638fae697/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:44.208 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 781637 Apr 22 06:26 /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:44.208 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:44.227 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:44.245 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T06:26:44.289 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 on all hosts... 2024-04-22T06:26:44.289 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:26:44.291 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:26:44.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 pull 2024-04-22T06:26:45.867 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:26:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:26:45.975 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:27:19.384 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:27:19.385 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-04-22T06:27:29.740 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-22T06:27:32.482 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-22T06:27:32.482 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev)", 2024-04-22T06:27:32.482 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61", 2024-04-22T06:27:32.483 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-04-22T06:27:32.483 INFO:teuthology.orchestra.run.smithi177.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2" 2024-04-22T06:27:32.483 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-04-22T06:27:32.483 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-22T06:27:32.506 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-04-22T06:27:32.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-04-22T06:27:32.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-04-22T06:27:32.598 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-04-22T06:27:32.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-04-22T06:27:32.651 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-04-22T06:27:32.675 INFO:tasks.cephadm:Writing seed config... 2024-04-22T06:27:32.676 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-22T06:27:32.676 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-22T06:27:32.676 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-22T06:27:32.676 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-22T06:27:32.676 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-22T06:27:32.677 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-22T06:27:32.678 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-22T06:27:32.679 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-22T06:27:32.680 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:27:32.680 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T06:27:32.697 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 48924e24-0071-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-22T06:27:32.698 DEBUG:teuthology.orchestra.run.smithi105:mon.a> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T06:27:32.740 DEBUG:teuthology.orchestra.run.smithi105:mgr.x> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T06:27:32.784 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T06:27:32.785 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 -v bootstrap --fsid 48924e24-0071-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.105 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:27:32.975 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:27:32.975 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697', '-v', 'bootstrap', '--fsid', '48924e24-0071-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.105', '--skip-admin-label'] 2024-04-22T06:27:32.976 INFO:teuthology.orchestra.run.smithi105.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-22T06:27:32.976 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying podman|docker is present... 2024-04-22T06:27:33.000 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:27:33.000 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying lvm2 is present... 2024-04-22T06:27:33.000 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying time synchronization is in place... 2024-04-22T06:27:33.007 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:27:33.008 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:27:33.013 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:27:33.014 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:27:33.020 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-04-22T06:27:33.025 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-04-22T06:27:33.026 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:27:33.026 INFO:teuthology.orchestra.run.smithi105.stdout:Repeating the final host check... 2024-04-22T06:27:33.050 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-22T06:27:33.050 INFO:teuthology.orchestra.run.smithi105.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-22T06:27:33.050 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl is present 2024-04-22T06:27:33.050 INFO:teuthology.orchestra.run.smithi105.stdout:lvcreate is present 2024-04-22T06:27:33.055 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-22T06:27:33.055 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T06:27:33.061 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-22T06:27:33.061 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:27:33.067 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-04-22T06:27:33.073 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-04-22T06:27:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-04-22T06:27:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:Host looks OK 2024-04-22T06:27:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:Cluster fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:Acquiring lock 140287322025120 on /run/cephadm/48924e24-0071-11ef-bc93-c7b262605968.lock 2024-04-22T06:27:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:Lock 140287322025120 acquired on /run/cephadm/48924e24-0071-11ef-bc93-c7b262605968.lock 2024-04-22T06:27:33.075 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 3300 ... 2024-04-22T06:27:33.075 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 6789 ... 2024-04-22T06:27:33.075 INFO:teuthology.orchestra.run.smithi105.stdout:Base mon IP(s) is [172.21.15.105:3300, 172.21.15.105:6789], mon addrv is [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-04-22T06:27:33.078 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.105 metric 100 2024-04-22T06:27:33.078 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.105 metric 100 2024-04-22T06:27:33.080 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T06:27:33.080 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T06:27:33.080 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 fe80::a72a:fe01:74f9:3d7c/64 scope link noprefixroute 2024-04-22T06:27:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-22T06:27:33.084 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-04-22T06:27:33.084 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-04-22T06:27:33.084 INFO:teuthology.orchestra.run.smithi105.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-22T06:27:33.084 INFO:teuthology.orchestra.run.smithi105.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T06:27:33.085 INFO:teuthology.orchestra.run.smithi105.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:27:33.519 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697... 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Getting image source signatures 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:3f93b098358ed33bdcfea942070678da0220cbea06d5b0225ae8923eb855cc02 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying config sha256:1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 2024-04-22T06:27:33.520 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-22T06:27:34.373 INFO:teuthology.orchestra.run.smithi105.stdout:ceph: stdout ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:27:34.374 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph version: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:27:34.374 INFO:teuthology.orchestra.run.smithi105.stdout:Extracting ceph user uid/gid from container image... 2024-04-22T06:27:35.337 INFO:teuthology.orchestra.run.smithi105.stdout:stat: stdout 167 167 2024-04-22T06:27:35.338 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial keys... 2024-04-22T06:27:36.183 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBXAyZmSqkXJRAAfVzt8RsrTBoLZ7FlAwwIDg== 2024-04-22T06:27:37.027 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBYAyZm4hzJHRAAkMupe0pscfLpAazVhOr+hg== 2024-04-22T06:27:37.952 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQBZAyZmGhhCFhAATX9KxwzDaQJ+RIy1J7RCsQ== 2024-04-22T06:27:37.953 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial monmap... 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool for a [v2:172.21.15.105:3300,v1:172.21.15.105:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = quincy 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: set fsid to 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:27:39.253 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mon... 2024-04-22T06:27:48.826 INFO:teuthology.orchestra.run.smithi105.stdout:create mon.a on 2024-04-22T06:27:49.387 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-22T06:27:49.681 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T06:27:50.178 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-48924e24-0071-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968.target. 2024-04-22T06:27:50.178 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-48924e24-0071-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968.target. 2024-04-22T06:27:50.408 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:27:50.408 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T06:27:50.561 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968.target.wants/ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968@.service. 2024-04-22T06:27:50.900 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:50 smithi105 podman[27523]: 2024-04-22 06:27:50.71811708 +0000 UTC m=+0.018297581 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:27:51.085 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:27:51.085 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-22T06:27:51.092 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:27:51.092 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:27:51.092 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-22T06:27:51.092 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:27:51.092 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon to start... 2024-04-22T06:27:51.093 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon... 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:50 smithi105 podman[27523]: 2024-04-22 06:27:50.995993125 +0000 UTC m=+0.296173616 container init ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:50 smithi105 podman[27523]: 2024-04-22 06:27:50.998780106 +0000 UTC m=+0.298960606 container start ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:51 smithi105 ceph-mon[27541]: mkfs 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:51 smithi105 ceph-mon[27541]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:51 smithi105 bash[27523]: ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d 2024-04-22T06:27:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:51 smithi105 systemd[1]: Started Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: monmap epoch 1 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: last_changed 2024-04-22T06:27:38.293503+0000 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: min_mon_release 19 (squid) 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: election_strategy: 1 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: fsmap 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:27:52.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: mgrmap e1: no daemons active 2024-04-22T06:27:52.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:52 smithi105 ceph-mon[27541]: from='client.? 172.21.15.105:0/2777806653' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout cluster: 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout id: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout services: 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.560921s) 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-22T06:27:52.161 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout data: 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pgs: 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.523+0000 7f093ff62700 1 Processor -- start 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.523+0000 7f093ff62700 1 -- start start 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093ff62700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093ff62700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f093807be60 con 0x7f0938109030 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093dcfe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093dcfe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40760/0 (socket says 172.21.15.105:40760) 2024-04-22T06:27:52.162 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093dcfe700 1 -- 172.21.15.105:0/349761847 learned_addr learned my addr 172.21.15.105:0/349761847 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093dcfe700 1 -- 172.21.15.105:0/349761847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093810b960 con 0x7f0938109030 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093dcfe700 1 --2- 172.21.15.105:0/349761847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09280098a0 tx=0x7f0928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f8a66a6f09f4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.524+0000 7f093ccfc700 1 -- 172.21.15.105:0/349761847 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0928004160 con 0x7f0938109030 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ccfc700 1 -- 172.21.15.105:0/349761847 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0928004300 con 0x7f0938109030 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ccfc700 1 -- 172.21.15.105:0/349761847 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09280036a0 con 0x7f0938109030 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 -- 172.21.15.105:0/349761847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 msgr2=0x7f093810b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 --2- 172.21.15.105:0/349761847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09280098a0 tx=0x7f0928009bb0 comp rx=0 tx=0).stop 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 -- 172.21.15.105:0/349761847 shutdown_connections 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 --2- 172.21.15.105:0/349761847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f093810b420 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 -- 172.21.15.105:0/349761847 >> 172.21.15.105:0/349761847 conn(0x7f0938102a80 msgr2=0x7f0938104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 -- 172.21.15.105:0/349761847 shutdown_connections 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.525+0000 7f093ff62700 1 -- 172.21.15.105:0/349761847 wait complete. 2024-04-22T06:27:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093ff62700 1 Processor -- start 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093ff62700 1 -- start start 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093ff62700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093ff62700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f093810bdd0 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093dcfe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093dcfe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40768/0 (socket says 172.21.15.105:40768) 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093dcfe700 1 -- 172.21.15.105:0/2777806653 learned_addr learned my addr 172.21.15.105:0/2777806653 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.526+0000 7f093dcfe700 1 -- 172.21.15.105:0/2777806653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0938108df0 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f093dcfe700 1 --2- 172.21.15.105:0/2777806653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0928000c00 tx=0x7f0928004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0928003db0 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0928011630 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0938106fe0 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0928011950 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0938107520 con 0x7f0938109030 2024-04-22T06:27:52.164 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f092801a460 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.527+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f092801b780 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.528+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09380640a0 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.530+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f092801ba20 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.600+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0938195700 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.600+0000 7f092effd700 1 -- 172.21.15.105:0/2777806653 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f092801bc00 con 0x7f0938109030 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 msgr2=0x7f09381088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 --2- 172.21.15.105:0/2777806653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0928000c00 tx=0x7f0928004750 comp rx=0 tx=0).stop 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 shutdown_connections 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 --2- 172.21.15.105:0/2777806653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0938109030 0x7f09381088b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 >> 172.21.15.105:0/2777806653 conn(0x7f0938102a80 msgr2=0x7f0938104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 shutdown_connections 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:51.601+0000 7f093ff62700 1 -- 172.21.15.105:0/2777806653 wait complete. 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:mon is available 2024-04-22T06:27:52.165 INFO:teuthology.orchestra.run.smithi105.stdout:Assimilating anything we can from ceph.conf... 2024-04-22T06:27:53.195 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:53 smithi105 ceph-mon[27541]: from='client.? 172.21.15.105:0/2410044896' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:27:53.195 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:53 smithi105 ceph-mon[27541]: from='client.? 172.21.15.105:0/2410044896' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:27:53.204 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.569+0000 7f552d3b4700 1 Processor -- start 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.569+0000 7f552d3b4700 1 -- start start 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.569+0000 7f552d3b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.569+0000 7f552d3b4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5528107e40 con 0x7f5528107450 2024-04-22T06:27:53.205 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40776/0 (socket says 172.21.15.105:40776) 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5526ffd700 1 -- 172.21.15.105:0/3429163087 learned_addr learned my addr 172.21.15.105:0/3429163087 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5526ffd700 1 -- 172.21.15.105:0/3429163087 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5528108650 con 0x7f5528107450 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5526ffd700 1 --2- 172.21.15.105:0/3429163087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f551c0098a0 tx=0x7f551c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1a4b6cd234f2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5525ffb700 1 -- 172.21.15.105:0/3429163087 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c004160 con 0x7f5528107450 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5525ffb700 1 -- 172.21.15.105:0/3429163087 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c004300 con 0x7f5528107450 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.570+0000 7f5525ffb700 1 -- 172.21.15.105:0/3429163087 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c0036a0 con 0x7f5528107450 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 -- 172.21.15.105:0/3429163087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 msgr2=0x7f5528107870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 --2- 172.21.15.105:0/3429163087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f551c0098a0 tx=0x7f551c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 -- 172.21.15.105:0/3429163087 shutdown_connections 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 --2- 172.21.15.105:0/3429163087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f5528107870 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 -- 172.21.15.105:0/3429163087 >> 172.21.15.105:0/3429163087 conn(0x7f5528102a10 msgr2=0x7f5528104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:53.206 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 -- 172.21.15.105:0/3429163087 shutdown_connections 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 -- 172.21.15.105:0/3429163087 wait complete. 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.571+0000 7f552d3b4700 1 Processor -- start 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f552d3b4700 1 -- start start 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f552d3b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f552d3b4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5528108b80 con 0x7f5528107450 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40784/0 (socket says 172.21.15.105:40784) 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f5526ffd700 1 -- 172.21.15.105:0/2410044896 learned_addr learned my addr 172.21.15.105:0/2410044896 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f5526ffd700 1 -- 172.21.15.105:0/2410044896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552819e400 con 0x7f5528107450 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f5526ffd700 1 --2- 172.21.15.105:0/2410044896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f551c000c00 tx=0x7f551c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c003db0 con 0x7f5528107450 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.572+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c011630 con 0x7f5528107450 2024-04-22T06:27:53.207 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.573+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552819e690 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.573+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c011950 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.573+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281a1380 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.573+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f551c01a460 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.573+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f551c01b780 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.574+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55280640a0 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.576+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f551c031020 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.643+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f552802e280 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.646+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+494 (secure 0 0 0) 0x7f551c01b9d0 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.647+0000 7f550ffff700 1 -- 172.21.15.105:0/2410044896 <== mon.0 v2:172.21.15.105:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f551c01ad20 con 0x7f5528107450 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 msgr2=0x7f552819dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 --2- 172.21.15.105:0/2410044896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f551c000c00 tx=0x7f551c004750 comp rx=0 tx=0).stop 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 shutdown_connections 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 --2- 172.21.15.105:0/2410044896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5528107450 0x7f552819dec0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:53.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 >> 172.21.15.105:0/2410044896 conn(0x7f5528102a10 msgr2=0x7f5528190750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:53.209 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 shutdown_connections 2024-04-22T06:27:53.209 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:52.648+0000 7f552d3b4700 1 -- 172.21.15.105:0/2410044896 wait complete. 2024-04-22T06:27:53.209 INFO:teuthology.orchestra.run.smithi105.stdout:Generating new minimal ceph.conf... 2024-04-22T06:27:54.144 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 ceph-mon[27541]: from='client.? 172.21.15.105:0/2872112733' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:27:54.150 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f696e92a700 1 Processor -- start 2024-04-22T06:27:54.154 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f696e92a700 1 -- start start 2024-04-22T06:27:54.154 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f696e92a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:54.154 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f696e92a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f696807c8e0 con 0x7f696807de60 2024-04-22T06:27:54.154 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f6967fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:54.154 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f6967fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40800/0 (socket says 172.21.15.105:40800) 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.588+0000 7f6967fff700 1 -- 172.21.15.105:0/2874699137 learned_addr learned my addr 172.21.15.105:0/2874699137 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.589+0000 7f6967fff700 1 -- 172.21.15.105:0/2874699137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696807ca60 con 0x7f696807de60 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.589+0000 7f6967fff700 1 --2- 172.21.15.105:0/2874699137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f69580098a0 tx=0x7f6958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9435c11a59ed67 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.589+0000 7f6966ffd700 1 -- 172.21.15.105:0/2874699137 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6958004160 con 0x7f696807de60 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.589+0000 7f6966ffd700 1 -- 172.21.15.105:0/2874699137 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6958004300 con 0x7f696807de60 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.589+0000 7f6966ffd700 1 -- 172.21.15.105:0/2874699137 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6958011af0 con 0x7f696807de60 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- 172.21.15.105:0/2874699137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 msgr2=0x7f696807c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 --2- 172.21.15.105:0/2874699137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f69580098a0 tx=0x7f6958009bb0 comp rx=0 tx=0).stop 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- 172.21.15.105:0/2874699137 shutdown_connections 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 --2- 172.21.15.105:0/2874699137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696807c310 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- 172.21.15.105:0/2874699137 >> 172.21.15.105:0/2874699137 conn(0x7f6968103c40 msgr2=0x7f6968106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- 172.21.15.105:0/2874699137 shutdown_connections 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- 172.21.15.105:0/2874699137 wait complete. 2024-04-22T06:27:54.155 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 Processor -- start 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.590+0000 7f696e92a700 1 -- start start 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f696e92a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f696e92a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f696810ead0 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f6967fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f6967fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40802/0 (socket says 172.21.15.105:40802) 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f6967fff700 1 -- 172.21.15.105:0/2872112733 learned_addr learned my addr 172.21.15.105:0/2872112733 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f6967fff700 1 -- 172.21.15.105:0/2872112733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696819ec10 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f6967fff700 1 --2- 172.21.15.105:0/2872112733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6958009fb0 tx=0x7f6958011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69580036a0 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f695801a430 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696819eea0 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6958019550 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69681a1b90 con 0x7f696807de60 2024-04-22T06:27:54.156 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.591+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6958019be0 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.592+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6958024730 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.592+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69680640a0 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.594+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6958043e50 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.663+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f69681a20d0 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.664+0000 7f69657fa700 1 -- 172.21.15.105:0/2872112733 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f6958032030 con 0x7f696807de60 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 msgr2=0x7f696819e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 --2- 172.21.15.105:0/2872112733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6958009fb0 tx=0x7f6958011940 comp rx=0 tx=0).stop 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 shutdown_connections 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 --2- 172.21.15.105:0/2872112733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f696807de60 0x7f696819e6d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 >> 172.21.15.105:0/2872112733 conn(0x7f6968103c40 msgr2=0x7f69681047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 shutdown_connections 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:53.665+0000 7f696e92a700 1 -- 172.21.15.105:0/2872112733 wait complete. 2024-04-22T06:27:54.157 INFO:teuthology.orchestra.run.smithi105.stdout:Restarting the monitor... 2024-04-22T06:27:54.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 systemd[1]: Stopping Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:27:54.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a[27537]: 2024-04-22T06:27:54.390+0000 7f0d12e15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:27:54.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a[27537]: 2024-04-22T06:27:54.390+0000 7f0d12e15700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T06:27:54.900 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 podman[27788]: 2024-04-22 06:27:54.645832267 +0000 UTC m=+0.339166171 container died ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:27:55.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 podman[27788]: 2024-04-22 06:27:54.984525326 +0000 UTC m=+0.677859220 container cleanup ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:27:55.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:54 smithi105 bash[27788]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:27:55.597 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 podman[27802]: 2024-04-22 06:27:55.177059643 +0000 UTC m=+0.529023650 container remove ba365f972478f8e89a3fb83792a94891fda953b156b78ba42370c443c765077d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-22T06:27:55.598 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T06:27:55.598 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 systemd[1]: Stopped Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:27:55.598 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 systemd[1]: Starting Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:27:55.870 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 podman[27897]: 2024-04-22 06:27:55.501798099 +0000 UTC m=+0.017785457 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:27:55.870 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 podman[27897]: 2024-04-22 06:27:55.695029947 +0000 UTC m=+0.211017275 container create 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:27:55.870 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 podman[27897]: 2024-04-22 06:27:55.872093197 +0000 UTC m=+0.388080525 container init 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:27:55.941 INFO:teuthology.orchestra.run.smithi105.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-22T06:27:56.153 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 podman[27897]: 2024-04-22 06:27:55.875029602 +0000 UTC m=+0.391016930 container start 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: pidfile_write: ignore empty --pid-file 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: load: jerasure load: lrc 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Git sha 0 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: DB SUMMARY 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: DB Session ID: FZGLKGWJ49EZTFF3T2OY 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:27:56.154 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105502 ; 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.env: 0x5603c8f34cc0 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.info_log: 0x5603ca8a3be0 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.statistics: (nil) 2024-04-22T06:27:56.155 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.use_fsync: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.db_log_dir: 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.wal_dir: 2024-04-22T06:27:56.156 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.write_buffer_manager: 0x5603ca8370e0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:27:56.157 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.unordered_write: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.row_cache: None 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.wal_filter: None 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.wal_compression: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:27:56.159 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:27:56.159 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:27:56.160 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_open_files: -1 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:27:56.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Compression algorithms supported: 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kZSTD supported: 0 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:27:56.162 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.merge_operator: 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_filter: None 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5603ca8a3dc0) 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: index_type: 0 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: data_block_index_type: 0 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: index_shortening: 1 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:27:56.163 INFO:journalctl@ceph.mon.a.smithi105.stdout: checksum: 4 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: no_block_cache: 0 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_cache: 0x5603ca8691f0 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_cache_options: 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: capacity : 536870912 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: num_shard_bits : 4 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: strict_capacity_limit : 0 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_cache_compressed: (nil) 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: persistent_cache: (nil) 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_size: 4096 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_size_deviation: 10 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_restart_interval: 16 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: index_block_restart_interval: 1 2024-04-22T06:27:56.164 INFO:journalctl@ceph.mon.a.smithi105.stdout: metadata_block_size: 4096 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: partition_filters: 0 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: use_delta_encoding: 1 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: filter_policy: bloomfilter 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: whole_key_filtering: 1 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: verify_compression: 0 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: format_version: 5 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: enable_index_compression: 1 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: block_align: 0 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: max_auto_readahead_size: 262144 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: prepopulate_block_cache: 0 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:27:56.165 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression: NoCompression 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.num_levels: 7 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:27:56.166 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:27:56.167 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:27:56.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:27:56.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.ttl: 2592000 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:27:56.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0a1dd162-c4c4-4dfe-844e-c387fc2b2e6d 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767275903581, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767275904556, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98481, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14272, "raw_average_key_size": 49, "raw_value_size": 90745, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767275, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0a1dd162-c4c4-4dfe-844e-c387fc2b2e6d", "db_session_id": "FZGLKGWJ49EZTFF3T2OY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:27:56.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767275904636, "job": 1, "event": "recovery_finished"} 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5603ca97e000 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: DB pointer 0x5603ca968000 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: ** DB Stats ** 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.172 INFO:journalctl@ceph.mon.a.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: L0 2/0 100.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Sum 2/0 100.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: ** Compaction Stats [default] ** 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:27:56.173 INFO:journalctl@ceph.mon.a.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: Cumulative compaction: 0.00 GB write, 32.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: Interval compaction: 0.00 GB write, 32.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: Block cache BinnedLRUCache@0x5603ca8691f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: starting mon.a rank 0 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???) e1 preinit fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:56.174 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).mds e1 new map 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).mds e1 print_map 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: e1 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: legacy client fscid: -1 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout: No filesystems configured 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 bash[27897]: 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 systemd[1]: Started Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T06:27:56.175 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: monmap epoch 1 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: last_changed 2024-04-22T06:27:38.293503+0000 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: min_mon_release 19 (squid) 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: election_strategy: 1 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: fsmap 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T06:27:56.176 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:55 smithi105 ceph-mon[27916]: mgrmap e1: no daemons active 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7feda2808700 1 Processor -- start 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7feda2808700 1 -- start start 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7feda2808700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7feda2808700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c07bdc0 con 0x7fed9c1090b0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7fed9bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7fed9bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60962/0 (socket says 172.21.15.105:60962) 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.377+0000 7fed9bfff700 1 -- 172.21.15.105:0/723667111 learned_addr learned my addr 172.21.15.105:0/723667111 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.378+0000 7fed9bfff700 1 -- 172.21.15.105:0/723667111 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c10b9e0 con 0x7fed9c1090b0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.378+0000 7fed9bfff700 1 --2- 172.21.15.105:0/723667111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0098a0 tx=0x7fed8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234fa01973f1fb64 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.378+0000 7fed9affd700 1 -- 172.21.15.105:0/723667111 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed8c004160 con 0x7fed9c1090b0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.378+0000 7fed9affd700 1 -- 172.21.15.105:0/723667111 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed8c004300 con 0x7fed9c1090b0 2024-04-22T06:27:56.984 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.378+0000 7fed9affd700 1 -- 172.21.15.105:0/723667111 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed8c011af0 con 0x7fed9c1090b0 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- 172.21.15.105:0/723667111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 msgr2=0x7fed9c10b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 --2- 172.21.15.105:0/723667111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0098a0 tx=0x7fed8c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- 172.21.15.105:0/723667111 shutdown_connections 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 --2- 172.21.15.105:0/723667111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c10b4a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- 172.21.15.105:0/723667111 >> 172.21.15.105:0/723667111 conn(0x7fed9c102a40 msgr2=0x7fed9c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- 172.21.15.105:0/723667111 shutdown_connections 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- 172.21.15.105:0/723667111 wait complete. 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 Processor -- start 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.379+0000 7feda2808700 1 -- start start 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7feda2808700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7feda2808700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c10c8e0 con 0x7fed9c1090b0 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed9bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed9bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60966/0 (socket says 172.21.15.105:60966) 2024-04-22T06:27:56.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed9bfff700 1 -- 172.21.15.105:0/2175509720 learned_addr learned my addr 172.21.15.105:0/2175509720 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed9bfff700 1 -- 172.21.15.105:0/2175509720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c19ca20 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed9bfff700 1 --2- 172.21.15.105:0/2175509720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed8c011ac0 tx=0x7fed8c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed8c003970 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed8c003b10 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9c19ccb0 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.380+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed8c024610 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.381+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9c19f9a0 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.381+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed8c024c20 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.381+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed8c019700 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.381+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed9c0640a0 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.384+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fed8c043e50 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.453+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fed9c19fb70 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.454+0000 7fed997fa700 1 -- 172.21.15.105:0/2175509720 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fed8c032030 con 0x7fed9c1090b0 2024-04-22T06:27:56.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 msgr2=0x7fed9c19c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 --2- 172.21.15.105:0/2175509720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fed8c011ac0 tx=0x7fed8c011ba0 comp rx=0 tx=0).stop 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 shutdown_connections 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 --2- 172.21.15.105:0/2175509720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed9c1090b0 0x7fed9c19c4e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 >> 172.21.15.105:0/2175509720 conn(0x7fed9c102a40 msgr2=0x7fed9c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 shutdown_connections 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:56.455+0000 7feda2808700 1 -- 172.21.15.105:0/2175509720 wait complete. 2024-04-22T06:27:56.987 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-22T06:27:57.001 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:27:57.002 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mgr... 2024-04-22T06:27:57.002 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-22T06:27:57.002 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-22T06:27:57.254 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:27:57.255 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-22T06:27:57.397 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968.target.wants/ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-48924e24-0071-11ef-bc93-c7b262605968@.service. 2024-04-22T06:27:57.413 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 systemd[1]: Starting Ceph mgr.x for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:27:57.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:57 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2175509720' entity='client.admin' 2024-04-22T06:27:57.669 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 podman[28133]: 2024-04-22 06:27:57.531914103 +0000 UTC m=+0.017427062 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:27:57.669 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 podman[28133]: 2024-04-22 06:27:57.630654989 +0000 UTC m=+0.116167937 container create f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-22T06:27:57.902 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:27:57.902 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-22T06:27:57.909 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:27:57.909 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:27:57.909 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-22T06:27:57.909 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-04-22T06:27:57.917 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:27:57.918 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr to start... 2024-04-22T06:27:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr... 2024-04-22T06:27:58.050 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 podman[28133]: 2024-04-22 06:27:57.824575303 +0000 UTC m=+0.310088252 container init f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:27:58.050 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 podman[28133]: 2024-04-22 06:27:57.826987088 +0000 UTC m=+0.312500038 container start f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:27:58.050 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 bash[28133]: f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 2024-04-22T06:27:58.050 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 systemd[1]: Started Ceph mgr.x for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:27:58.050 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:57 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:57.979+0000 7f5bbeeae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:27:58.359 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:58 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:58.167+0000 7f5bbeeae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:27:58.360 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:58 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:58.270+0000 7f5bbeeae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:27:58.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:27:58 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/669660073' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:27:58.669 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:58 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:58.604+0000 7f5bbeeae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:27:58.979 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:58 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:58.691+0000 7f5bbeeae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:27:58.982 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.985 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:27:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:27:58.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:27:58.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:27:58.987 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:27:58.995 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:27:58.996 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:27:51.041562+0000", 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.369+0000 7f25ad467700 1 Processor -- start 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.369+0000 7f25ad467700 1 -- start start 2024-04-22T06:27:58.997 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.369+0000 7f25ad467700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.369+0000 7f25ad467700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f25a807be40 con 0x7f25a8109050 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.369+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32772/0 (socket says 172.21.15.105:32772) 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a6ffd700 1 -- 172.21.15.105:0/195081349 learned_addr learned my addr 172.21.15.105:0/195081349 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a6ffd700 1 -- 172.21.15.105:0/195081349 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a810b980 con 0x7f25a8109050 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a6ffd700 1 --2- 172.21.15.105:0/195081349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f259c0098a0 tx=0x7f259c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f695c6f3ee83d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a5ffb700 1 -- 172.21.15.105:0/195081349 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f259c004160 con 0x7f25a8109050 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a5ffb700 1 -- 172.21.15.105:0/195081349 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f259c004300 con 0x7f25a8109050 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.370+0000 7f25a5ffb700 1 -- 172.21.15.105:0/195081349 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f259c011af0 con 0x7f25a8109050 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 -- 172.21.15.105:0/195081349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 msgr2=0x7f25a810b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 --2- 172.21.15.105:0/195081349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f259c0098a0 tx=0x7f259c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 -- 172.21.15.105:0/195081349 shutdown_connections 2024-04-22T06:27:58.998 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 --2- 172.21.15.105:0/195081349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a810b440 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 -- 172.21.15.105:0/195081349 >> 172.21.15.105:0/195081349 conn(0x7f25a8102a80 msgr2=0x7f25a8104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 -- 172.21.15.105:0/195081349 shutdown_connections 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 -- 172.21.15.105:0/195081349 wait complete. 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.371+0000 7f25ad467700 1 Processor -- start 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25ad467700 1 -- start start 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25ad467700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25ad467700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f25a810c8f0 con 0x7f25a8109050 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25a6ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32776/0 (socket says 172.21.15.105:32776) 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25a6ffd700 1 -- 172.21.15.105:0/669660073 learned_addr learned my addr 172.21.15.105:0/669660073 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25a6ffd700 1 -- 172.21.15.105:0/669660073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a819ca10 con 0x7f25a8109050 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.372+0000 7f25a6ffd700 1 --2- 172.21.15.105:0/669660073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f259c009fb0 tx=0x7f259c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f259c0036a0 con 0x7f25a8109050 2024-04-22T06:27:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f259c01a430 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f259c019550 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a819cca0 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a819f990 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f259c019b90 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.373+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f259c0248a0 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.374+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a80640a0 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.376+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f259c043e50 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.446+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f25a819fed0 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.446+0000 7f258ffff700 1 -- 172.21.15.105:0/669660073 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f259c032020 con 0x7f25a8109050 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 msgr2=0x7f25a819c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 --2- 172.21.15.105:0/669660073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f259c009fb0 tx=0x7f259c011940 comp rx=0 tx=0).stop 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 shutdown_connections 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 --2- 172.21.15.105:0/669660073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f25a8109050 0x7f25a819c4d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 >> 172.21.15.105:0/669660073 conn(0x7f25a8102a80 msgr2=0x7f25a81035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:27:59.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 shutdown_connections 2024-04-22T06:27:59.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:27:58.448+0000 7f25ad467700 1 -- 172.21.15.105:0/669660073 wait complete. 2024-04-22T06:27:59.001 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (1/15)... 2024-04-22T06:28:00.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:27:59 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:27:59.904+0000 7f5bbeeae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:28:00.169 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:00.068+0000 7f5bbeeae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:00.525 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:00.236+0000 7f5bbeeae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:28:00.525 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:00.525+0000 7f5bbeeae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:28:00.817 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:00.617+0000 7f5bbeeae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:28:00.817 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:00.818+0000 7f5bbeeae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:28:01.112 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:01 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:01.012+0000 7f5bbeeae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:28:01.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3425288357' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:28:01.669 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:01 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:01.628+0000 7f5bbeeae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:28:02.099 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:02.099 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:02.099 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:28:02.099 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:28:02.100 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:28:02.101 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:28:02.108 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:28:02.108 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:28:02.108 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:28:02.109 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:02.110 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:27:51.041562+0000", 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f3218e73700 1 Processor -- start 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f3218e73700 1 -- start start 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f3218e73700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f3218e73700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f321407bde0 con 0x7f3214109080 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f321259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f321259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32778/0 (socket says 172.21.15.105:32778) 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.433+0000 7f321259c700 1 -- 172.21.15.105:0/3875577385 learned_addr learned my addr 172.21.15.105:0/3875577385 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.434+0000 7f321259c700 1 -- 172.21.15.105:0/3875577385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321410b9b0 con 0x7f3214109080 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.434+0000 7f321259c700 1 --2- 172.21.15.105:0/3875577385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2f59fecac1047e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.434+0000 7f321159a700 1 -- 172.21.15.105:0/3875577385 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204004160 con 0x7f3214109080 2024-04-22T06:28:02.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.434+0000 7f321159a700 1 -- 172.21.15.105:0/3875577385 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3204004300 con 0x7f3214109080 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.434+0000 7f321159a700 1 -- 172.21.15.105:0/3875577385 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204011af0 con 0x7f3214109080 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 -- 172.21.15.105:0/3875577385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 msgr2=0x7f321410b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 --2- 172.21.15.105:0/3875577385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f32040098a0 tx=0x7f3204009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 -- 172.21.15.105:0/3875577385 shutdown_connections 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 --2- 172.21.15.105:0/3875577385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321410b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 -- 172.21.15.105:0/3875577385 >> 172.21.15.105:0/3875577385 conn(0x7f3214102a10 msgr2=0x7f3214104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 -- 172.21.15.105:0/3875577385 shutdown_connections 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 -- 172.21.15.105:0/3875577385 wait complete. 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.435+0000 7f3218e73700 1 Processor -- start 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f3218e73700 1 -- start start 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f3218e73700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f3218e73700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f321410c8c0 con 0x7f3214109080 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f321259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f321259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32784/0 (socket says 172.21.15.105:32784) 2024-04-22T06:28:02.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f321259c700 1 -- 172.21.15.105:0/3425288357 learned_addr learned my addr 172.21.15.105:0/3425288357 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f321259c700 1 -- 172.21.15.105:0/3425288357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321419c9e0 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f321259c700 1 --2- 172.21.15.105:0/3425288357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3204009fb0 tx=0x7f3204011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32040036a0 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f320401a430 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3204019550 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321419cc70 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.436+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f321419f960 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.437+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3204019be0 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.437+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3204024730 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.437+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32140640a0 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.439+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3204043e50 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.511+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f321419fea0 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.511+0000 7f32037fe700 1 -- 172.21.15.105:0/3425288357 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3204032070 con 0x7f3214109080 2024-04-22T06:28:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.512+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 msgr2=0x7f321419c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.512+0000 7f3218e73700 1 --2- 172.21.15.105:0/3425288357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3204009fb0 tx=0x7f3204011940 comp rx=0 tx=0).stop 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.512+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 shutdown_connections 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.512+0000 7f3218e73700 1 --2- 172.21.15.105:0/3425288357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3214109080 0x7f321419c4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.512+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 >> 172.21.15.105:0/3425288357 conn(0x7f3214102a10 msgr2=0x7f3214103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.513+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 shutdown_connections 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:01.513+0000 7f3218e73700 1 -- 172.21.15.105:0/3425288357 wait complete. 2024-04-22T06:28:02.114 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (2/15)... 2024-04-22T06:28:02.347 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:02.055+0000 7f5bbeeae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:28:02.347 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:02.151+0000 7f5bbeeae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:28:02.347 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:02.348+0000 7f5bbeeae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:28:03.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:02.859+0000 7f5bbeeae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:28:03.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:02.949+0000 7f5bbeeae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:28:03.518 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:03.185+0000 7f5bbeeae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:28:03.519 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:03.315+0000 7f5bbeeae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:28:03.519 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:03.407+0000 7f5bbeeae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:28:03.519 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:03.519+0000 7f5bbeeae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:04.169 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:03.814+0000 7f5bbeeae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:28:04.520 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:04 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:04.426+0000 7f5bbeeae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:28:04.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/752715452' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:28:04.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:04 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:04.656+0000 7f5bbeeae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.123 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:28:05.124 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:28:05.125 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.126 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:28:05.127 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:27:51.041562+0000", 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f266990f700 1 Processor -- start 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f266990f700 1 -- start start 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f266990f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:05.128 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f266990f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2664107e70 con 0x7f2664107480 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f2662ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f2662ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38696/0 (socket says 172.21.15.105:38696) 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.521+0000 7f2662ffd700 1 -- 172.21.15.105:0/3034547221 learned_addr learned my addr 172.21.15.105:0/3034547221 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.522+0000 7f2662ffd700 1 -- 172.21.15.105:0/3034547221 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2664108680 con 0x7f2664107480 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.522+0000 7f2662ffd700 1 --2- 172.21.15.105:0/3034547221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f265800bb40 tx=0x7f265800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f7589e17da36de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.522+0000 7f2661ffb700 1 -- 172.21.15.105:0/3034547221 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f265800dbe0 con 0x7f2664107480 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.522+0000 7f2661ffb700 1 -- 172.21.15.105:0/3034547221 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f26580044c0 con 0x7f2664107480 2024-04-22T06:28:05.129 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.522+0000 7f2661ffb700 1 -- 172.21.15.105:0/3034547221 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2658010470 con 0x7f2664107480 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 -- 172.21.15.105:0/3034547221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 msgr2=0x7f26641078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 --2- 172.21.15.105:0/3034547221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f265800bb40 tx=0x7f265800be50 comp rx=0 tx=0).stop 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 -- 172.21.15.105:0/3034547221 shutdown_connections 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 --2- 172.21.15.105:0/3034547221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f26641078a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 -- 172.21.15.105:0/3034547221 >> 172.21.15.105:0/3034547221 conn(0x7f2664102a80 msgr2=0x7f2664104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 -- 172.21.15.105:0/3034547221 shutdown_connections 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.523+0000 7f266990f700 1 -- 172.21.15.105:0/3034547221 wait complete. 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f266990f700 1 Processor -- start 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f266990f700 1 -- start start 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f266990f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f266990f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2664062e80 con 0x7f2664107480 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f2662ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:05.130 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f2662ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38702/0 (socket says 172.21.15.105:38702) 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f2662ffd700 1 -- 172.21.15.105:0/752715452 learned_addr learned my addr 172.21.15.105:0/752715452 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.524+0000 7f2662ffd700 1 -- 172.21.15.105:0/752715452 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266419a7c0 con 0x7f2664107480 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f2662ffd700 1 --2- 172.21.15.105:0/752715452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f265800bfd0 tx=0x7f2658003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2658027070 con 0x7f2664107480 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2658010c00 con 0x7f2664107480 2024-04-22T06:28:05.131 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266419aa50 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2658022c10 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f266419d6f0 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f265802a070 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.525+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2658022e00 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.526+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26640640a0 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.530+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2658031030 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.601+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f26641937e0 con 0x7f2664107480 2024-04-22T06:28:05.132 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.602+0000 7f266890d700 1 -- 172.21.15.105:0/752715452 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f26641937e0 con 0x7f2664107480 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.602+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 msgr2=0x7f266419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.602+0000 7f266990f700 1 --2- 172.21.15.105:0/752715452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f265800bfd0 tx=0x7f2658003ff0 comp rx=0 tx=0).stop 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.603+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 shutdown_connections 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.603+0000 7f266990f700 1 --2- 172.21.15.105:0/752715452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2664107480 0x7f266419a280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.603+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 >> 172.21.15.105:0/752715452 conn(0x7f2664102a80 msgr2=0x7f26641908a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.603+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 shutdown_connections 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:04.603+0000 7f266990f700 1 -- 172.21.15.105:0/752715452 wait complete. 2024-04-22T06:28:05.133 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (3/15)... 2024-04-22T06:28:05.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: Activating manager daemon x 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: mgrmap e2: x(active, starting, since 0.00317557s) 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: Manager daemon x is now available 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' 2024-04-22T06:28:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:05 smithi105 ceph-mon[27916]: from='mgr.14100 172.21.15.105:0/3433915758' entity='mgr.x' 2024-04-22T06:28:06.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:06 smithi105 ceph-mon[27916]: mgrmap e3: x(active, since 1.00621s) 2024-04-22T06:28:07.813 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:07 smithi105 ceph-mon[27916]: mgrmap e4: x(active, since 2s) 2024-04-22T06:28:08.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "a" 2024-04-22T06:28:08.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-22T06:28:08.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-22T06:28:08.288 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-22T06:28:08.289 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-04-22T06:28:08.290 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-04-22T06:27:51.041562+0000", 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f648aa69700 1 Processor -- start 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f648aa69700 1 -- start start 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f648aa69700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f648aa69700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6484107220 con 0x7f6484106830 2024-04-22T06:28:08.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f6488805700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f6488805700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38770/0 (socket says 172.21.15.105:38770) 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f6488805700 1 -- 172.21.15.105:0/2084525461 learned_addr learned my addr 172.21.15.105:0/2084525461 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.527+0000 7f6488805700 1 -- 172.21.15.105:0/2084525461 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64841097c0 con 0x7f6484106830 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f6488805700 1 --2- 172.21.15.105:0/2084525461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64780098a0 tx=0x7f6478009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a933220c9336f7dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f64837fe700 1 -- 172.21.15.105:0/2084525461 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478004160 con 0x7f6484106830 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f64837fe700 1 -- 172.21.15.105:0/2084525461 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6478004300 con 0x7f6484106830 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f64837fe700 1 -- 172.21.15.105:0/2084525461 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6478011af0 con 0x7f6484106830 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 -- 172.21.15.105:0/2084525461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 msgr2=0x7f6484106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 --2- 172.21.15.105:0/2084525461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64780098a0 tx=0x7f6478009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 -- 172.21.15.105:0/2084525461 shutdown_connections 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 --2- 172.21.15.105:0/2084525461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484106c50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 -- 172.21.15.105:0/2084525461 >> 172.21.15.105:0/2084525461 conn(0x7f6484102160 msgr2=0x7f64841045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.528+0000 7f648aa69700 1 -- 172.21.15.105:0/2084525461 shutdown_connections 2024-04-22T06:28:08.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f648aa69700 1 -- 172.21.15.105:0/2084525461 wait complete. 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f648aa69700 1 Processor -- start 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f648aa69700 1 -- start start 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f648aa69700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f648aa69700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f648410cb70 con 0x7f6484106830 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f6488805700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f6488805700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38776/0 (socket says 172.21.15.105:38776) 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.529+0000 7f6488805700 1 -- 172.21.15.105:0/8977910 learned_addr learned my addr 172.21.15.105:0/8977910 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f6488805700 1 -- 172.21.15.105:0/8977910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6484109580 con 0x7f6484106830 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f6488805700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6478003f40 tx=0x7f6478011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64780038b0 con 0x7f6484106830 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6478003a50 con 0x7f6484106830 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6484107770 con 0x7f6484106830 2024-04-22T06:28:08.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f647801a5c0 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.530+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6484107cb0 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.531+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f647801a870 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.531+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64840640a0 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.531+0000 7f6481ffb700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 0x7f647403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.532+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f647804dba0 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.532+0000 7f6483fff700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 0x7f647403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.532+0000 7f6483fff700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 0x7f647403ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6470006fd0 tx=0x7f6470006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.537+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6478019c70 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.814+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6484107f30 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.814+0000 7f6481ffb700 1 -- 172.21.15.105:0/8977910 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f647804d5c0 con 0x7f6484106830 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 msgr2=0x7f647403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 0x7f647403ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6470006fd0 tx=0x7f6470006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:08.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 msgr2=0x7f6484109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6478003f40 tx=0x7f6478011890 comp rx=0 tx=0).stop 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 shutdown_connections 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f6474039560 0x7f647403ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 --2- 172.21.15.105:0/8977910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6484106830 0x7f6484109040 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.817+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 >> 172.21.15.105:0/8977910 conn(0x7f6484102160 msgr2=0x7f6484102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.818+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 shutdown_connections 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:07.818+0000 7f648aa69700 1 -- 172.21.15.105:0/8977910 wait complete. 2024-04-22T06:28:08.295 INFO:teuthology.orchestra.run.smithi105.stdout:mgr is available 2024-04-22T06:28:08.902 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:08 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/8977910' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T06:28:09.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-04-22T06:28:09.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f078ba26700 1 Processor -- start 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f078ba26700 1 -- start start 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f078ba26700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f078ba26700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f078407bea0 con 0x7f0784108fd0 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f07897c2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f07897c2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38792/0 (socket says 172.21.15.105:38792) 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.701+0000 7f07897c2700 1 -- 172.21.15.105:0/443040011 learned_addr learned my addr 172.21.15.105:0/443040011 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f07897c2700 1 -- 172.21.15.105:0/443040011 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078410b900 con 0x7f0784108fd0 2024-04-22T06:28:09.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f07897c2700 1 --2- 172.21.15.105:0/443040011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f07800098a0 tx=0x7f0780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ff12dab6644e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f077bfff700 1 -- 172.21.15.105:0/443040011 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0780004160 con 0x7f0784108fd0 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f077bfff700 1 -- 172.21.15.105:0/443040011 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0780004300 con 0x7f0784108fd0 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f077bfff700 1 -- 172.21.15.105:0/443040011 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0780011af0 con 0x7f0784108fd0 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f078ba26700 1 -- 172.21.15.105:0/443040011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 msgr2=0x7f078410b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.702+0000 7f078ba26700 1 --2- 172.21.15.105:0/443040011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f07800098a0 tx=0x7f0780009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- 172.21.15.105:0/443040011 shutdown_connections 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 --2- 172.21.15.105:0/443040011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078410b3c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- 172.21.15.105:0/443040011 >> 172.21.15.105:0/443040011 conn(0x7f0784102a40 msgr2=0x7f0784104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- 172.21.15.105:0/443040011 shutdown_connections 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- 172.21.15.105:0/443040011 wait complete. 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 Processor -- start 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- start start 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.703+0000 7f078ba26700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f078410c8d0 con 0x7f0784108fd0 2024-04-22T06:28:09.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f07897c2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f07897c2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38804/0 (socket says 172.21.15.105:38804) 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f07897c2700 1 -- 172.21.15.105:0/3700788920 learned_addr learned my addr 172.21.15.105:0/3700788920 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f07897c2700 1 -- 172.21.15.105:0/3700788920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078419ca40 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f07897c2700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0780009fb0 tx=0x7f0780011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07800036a0 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f078001a430 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078419ccd0 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0780019550 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.704+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078419f9c0 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.705+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f0780019800 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.705+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07840640a0 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.705+0000 7f077a7fc700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 0x7f077003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.705+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f078004d850 con 0x7f0784108fd0 2024-04-22T06:28:09.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.705+0000 7f0788fc1700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 0x7f077003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.706+0000 7f0788fc1700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 0x7f077003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0774006fd0 tx=0x7f0774006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.710+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f078002cc80 con 0x7f0784108fd0 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.903+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f078419fb90 con 0x7f0784108fd0 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.904+0000 7f077a7fc700 1 -- 172.21.15.105:0/3700788920 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7f078419fb90 con 0x7f0784108fd0 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 msgr2=0x7f077003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 0x7f077003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0774006fd0 tx=0x7f0774006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 msgr2=0x7f078419c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0780009fb0 tx=0x7f0780011940 comp rx=0 tx=0).stop 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 shutdown_connections 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f0770039560 0x7f077003ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 --2- 172.21.15.105:0/3700788920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0784108fd0 0x7f078419c500 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 >> 172.21.15.105:0/3700788920 conn(0x7f0784102a40 msgr2=0x7f07841035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.906+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 shutdown_connections 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:08.907+0000 7f078ba26700 1 -- 172.21.15.105:0/3700788920 wait complete. 2024-04-22T06:28:09.442 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling cephadm module... 2024-04-22T06:28:09.788 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:09 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3700788920' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T06:28:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:10 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/63529136' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T06:28:10.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setuser ceph since I am not root 2024-04-22T06:28:10.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setgroup ceph since I am not root 2024-04-22T06:28:10.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:10.836+0000 7f524bca6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:28:11.243 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:11.021+0000 7f524bca6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:28:11.244 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:11.123+0000 7f524bca6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bcc473700 1 Processor -- start 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bcc473700 1 -- start start 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bcc473700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bcc473700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc4107240 con 0x7f4bc4106850 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bca20f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bca20f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38818/0 (socket says 172.21.15.105:38818) 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.789+0000 7f4bca20f700 1 -- 172.21.15.105:0/2106033008 learned_addr learned my addr 172.21.15.105:0/2106033008 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.790+0000 7f4bca20f700 1 -- 172.21.15.105:0/2106033008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc4109880 con 0x7f4bc4106850 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.790+0000 7f4bca20f700 1 --2- 172.21.15.105:0/2106033008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00098a0 tx=0x7f4bc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d618672c5ffd129 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:11.251 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.790+0000 7f4bc920d700 1 -- 172.21.15.105:0/2106033008 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0004160 con 0x7f4bc4106850 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.790+0000 7f4bc920d700 1 -- 172.21.15.105:0/2106033008 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4bc0004300 con 0x7f4bc4106850 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.790+0000 7f4bc920d700 1 -- 172.21.15.105:0/2106033008 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0011af0 con 0x7f4bc4106850 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- 172.21.15.105:0/2106033008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 msgr2=0x7f4bc4106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 --2- 172.21.15.105:0/2106033008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00098a0 tx=0x7f4bc0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- 172.21.15.105:0/2106033008 shutdown_connections 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 --2- 172.21.15.105:0/2106033008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc4106c70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- 172.21.15.105:0/2106033008 >> 172.21.15.105:0/2106033008 conn(0x7f4bc41021e0 msgr2=0x7f4bc4104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- 172.21.15.105:0/2106033008 shutdown_connections 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- 172.21.15.105:0/2106033008 wait complete. 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 Processor -- start 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- start start 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bcc473700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc410cab0 con 0x7f4bc4106850 2024-04-22T06:28:11.252 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.791+0000 7f4bca20f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bca20f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38820/0 (socket says 172.21.15.105:38820) 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bca20f700 1 -- 172.21.15.105:0/63529136 learned_addr learned my addr 172.21.15.105:0/63529136 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bca20f700 1 -- 172.21.15.105:0/63529136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc419c9c0 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bca20f700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0003b20 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc419cc50 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4bc001a8c0 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4bc0024940 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.792+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc419f940 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4bc0022020 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc40640a0 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bbb7fe700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 0x7f4bb003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4bc004d5c0 con 0x7f4bc4106850 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bc9a0e700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 0x7f4bb003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.793+0000 7f4bc9a0e700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 0x7f4bb003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4006fd0 tx=0x7f4bb4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:09.798+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4bc00042c0 con 0x7f4bc4106850 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.050+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f4bc419fb10 con 0x7f4bc4106850 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.670+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f4bc419fb10 con 0x7f4bc4106850 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.671+0000 7f4bbb7fe700 1 -- 172.21.15.105:0/63529136 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4bc001aae0 con 0x7f4bc4106850 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 msgr2=0x7f4bb003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 0x7f4bb003ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4006fd0 tx=0x7f4bb4006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 msgr2=0x7f4bc419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0000c00 tx=0x7f4bc0011b70 comp rx=0 tx=0).stop 2024-04-22T06:28:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 shutdown_connections 2024-04-22T06:28:11.255 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7f4bb0039560 0x7f4bb003ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:11.255 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 --2- 172.21.15.105:0/63529136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bc4106850 0x7f4bc419c480 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:11.255 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 >> 172.21.15.105:0/63529136 conn(0x7f4bc41021e0 msgr2=0x7f4bc4102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:11.255 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 shutdown_connections 2024-04-22T06:28:11.255 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:10.673+0000 7f4bcc473700 1 -- 172.21.15.105:0/63529136 wait complete. 2024-04-22T06:28:11.529 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:11.440+0000 7f524bca6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:28:11.529 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:11.529+0000 7f524bca6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:28:11.886 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/63529136' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T06:28:11.886 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:11 smithi105 ceph-mon[27916]: mgrmap e5: x(active, since 6s) 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.681+0000 7ffabb609700 1 Processor -- start 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.681+0000 7ffabb609700 1 -- start start 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.681+0000 7ffabb609700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:12.534 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.681+0000 7ffabb609700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffab4063d20 con 0x7ffab4109030 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffab93a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffab93a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38844/0 (socket says 172.21.15.105:38844) 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffab93a5700 1 -- 172.21.15.105:0/3145161656 learned_addr learned my addr 172.21.15.105:0/3145161656 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffab93a5700 1 -- 172.21.15.105:0/3145161656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab410b960 con 0x7ffab4109030 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffab93a5700 1 --2- 172.21.15.105:0/3145161656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffab00098a0 tx=0x7ffab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc7beaf9252f824 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffaabfff700 1 -- 172.21.15.105:0/3145161656 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab0004160 con 0x7ffab4109030 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffaabfff700 1 -- 172.21.15.105:0/3145161656 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffab0004300 con 0x7ffab4109030 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.682+0000 7ffaabfff700 1 -- 172.21.15.105:0/3145161656 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab0011af0 con 0x7ffab4109030 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 -- 172.21.15.105:0/3145161656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 msgr2=0x7ffab410b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 --2- 172.21.15.105:0/3145161656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffab00098a0 tx=0x7ffab0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 -- 172.21.15.105:0/3145161656 shutdown_connections 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 --2- 172.21.15.105:0/3145161656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab410b420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 -- 172.21.15.105:0/3145161656 >> 172.21.15.105:0/3145161656 conn(0x7ffab4102a80 msgr2=0x7ffab4104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:12.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 -- 172.21.15.105:0/3145161656 shutdown_connections 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.683+0000 7ffabb609700 1 -- 172.21.15.105:0/3145161656 wait complete. 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffabb609700 1 Processor -- start 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffabb609700 1 -- start start 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffabb609700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffabb609700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffab410c920 con 0x7ffab4109030 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffab93a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffab93a5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38850/0 (socket says 172.21.15.105:38850) 2024-04-22T06:28:12.536 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffab93a5700 1 -- 172.21.15.105:0/2465541023 learned_addr learned my addr 172.21.15.105:0/2465541023 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.684+0000 7ffab93a5700 1 -- 172.21.15.105:0/2465541023 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab419c9f0 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffab93a5700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffab0011ac0 tx=0x7ffab0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab0003b20 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab419cc80 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffab001a8c0 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffab00248d0 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.685+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab419f970 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7ffab0022020 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab40640a0 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffaaa7fc700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 0x7ffaa003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffab004d550 con 0x7ffab4109030 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffab8ba4700 1 -- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 msgr2=0x7ffaa003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.686+0000 7ffab8ba4700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 0x7ffaa003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:28:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.694+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffab002cdf0 con 0x7ffab4109030 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.887+0000 7ffab8ba4700 1 -- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 msgr2=0x7ffaa003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.887+0000 7ffab8ba4700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 0x7ffaa003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.992+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ffab419fb40 con 0x7ffab4109030 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.992+0000 7ffaaa7fc700 1 -- 172.21.15.105:0/2465541023 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7ffab419fb40 con 0x7ffab4109030 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 msgr2=0x7ffaa003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:28:12.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 0x7ffaa003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 msgr2=0x7ffab419c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffab0011ac0 tx=0x7ffab0011ba0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 shutdown_connections 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7ffaa00395b0 0x7ffaa003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 --2- 172.21.15.105:0/2465541023 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffab4109030 0x7ffab419c4b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.994+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 >> 172.21.15.105:0/2465541023 conn(0x7ffab4102a80 msgr2=0x7ffab4103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.995+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 shutdown_connections 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:11.995+0000 7ffabb609700 1 -- 172.21.15.105:0/2465541023 wait complete. 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-04-22T06:28:12.539 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 5... 2024-04-22T06:28:12.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:12 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2465541023' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:28:12.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:12.733+0000 7f524bca6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:28:12.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:12.909+0000 7f524bca6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:13.169 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:13.077+0000 7f524bca6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:28:13.647 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:13.360+0000 7f524bca6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:28:13.647 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:13.450+0000 7f524bca6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:28:13.647 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:13.648+0000 7f524bca6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:28:13.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:13.837+0000 7f524bca6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:28:14.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:14.421+0000 7f524bca6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:28:15.121 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:14.832+0000 7f524bca6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:28:15.122 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:14.926+0000 7f524bca6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:28:15.122 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:15.122+0000 7f524bca6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:28:15.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:15.633+0000 7f524bca6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:28:15.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:15.723+0000 7f524bca6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:28:16.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:15.961+0000 7f524bca6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:28:16.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:16.089+0000 7f524bca6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:28:16.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:16.182+0000 7f524bca6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:28:16.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:16.294+0000 7f524bca6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:16.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:16.590+0000 7f524bca6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:28:17.441 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:17.178+0000 7f524bca6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:28:17.441 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:17.389+0000 7f524bca6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: Active manager daemon x restarted 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: Activating manager daemon x 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: mgrmap e6: x(active, starting, since 0.00316029s) 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: Manager daemon x is now available 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:17.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:17.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:17.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:28:17.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:17 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:28:18.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:18 smithi105 ceph-mon[27916]: Found migration_current of "None". Setting to last migration. 2024-04-22T06:28:18.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:18 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:18.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:18 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:18.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:18 smithi105 ceph-mon[27916]: mgrmap e7: x(active, since 1.0054s) 2024-04-22T06:28:19.017 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.936+0000 7fd65f6f4700 1 Processor -- start 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.936+0000 7fd65f6f4700 1 -- start start 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65f6f4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65f6f4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd65810a580 con 0x7fd658109b90 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65d490700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65d490700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38856/0 (socket says 172.21.15.105:38856) 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65d490700 1 -- 172.21.15.105:0/2386871394 learned_addr learned my addr 172.21.15.105:0/2386871394 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65d490700 1 -- 172.21.15.105:0/2386871394 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65810a700 con 0x7fd658109b90 2024-04-22T06:28:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd65d490700 1 --2- 172.21.15.105:0/2386871394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6480098a0 tx=0x7fd648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf45ec6a126aca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd64ffff700 1 -- 172.21.15.105:0/2386871394 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd648004160 con 0x7fd658109b90 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd64ffff700 1 -- 172.21.15.105:0/2386871394 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd648004300 con 0x7fd658109b90 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.937+0000 7fd64ffff700 1 -- 172.21.15.105:0/2386871394 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd648011af0 con 0x7fd658109b90 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2386871394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 msgr2=0x7fd658109fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2386871394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6480098a0 tx=0x7fd648009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2386871394 shutdown_connections 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2386871394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd658109fb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2386871394 >> 172.21.15.105:0/2386871394 conn(0x7fd658102210 msgr2=0x7fd658104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2386871394 shutdown_connections 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.938+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2386871394 wait complete. 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65f6f4700 1 Processor -- start 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65f6f4700 1 -- start start 2024-04-22T06:28:19.019 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65f6f4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65f6f4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd658107390 con 0x7fd658109b90 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65d490700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65d490700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38866/0 (socket says 172.21.15.105:38866) 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65d490700 1 -- 172.21.15.105:0/2459781207 learned_addr learned my addr 172.21.15.105:0/2459781207 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65d490700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6581096f0 con 0x7fd658109b90 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.939+0000 7fd65d490700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd648003f40 tx=0x7fd648011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.940+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6480038b0 con 0x7fd658109b90 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.940+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd648003a50 con 0x7fd658109b90 2024-04-22T06:28:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.940+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6581078e0 con 0x7fd658109b90 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.940+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd64801a5c0 con 0x7fd658109b90 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.940+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd658107e20 con 0x7fd658109b90 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd64801a870 con 0x7fd658109b90 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd64e7fc700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd64403c180 con 0x7fd6440395b0 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd64804da40 con 0x7fd658109b90 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd65cc8f700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:12.941+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:13.142+0000 7fd65cc8f700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:13.142+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:13.543+0000 7fd65cc8f700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:13.543+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:28:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:14.344+0000 7fd65cc8f700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:14.344+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:15.945+0000 7fd65cc8f700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2332170576 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:15.945+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:17.393+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fd64801abb0 con 0x7fd658109b90 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:17.393+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 msgr2=0x7fd64403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:17.393+0000 7fd64e7fc700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.394+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fd648018350 con 0x7fd658109b90 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.394+0000 7fd64e7fc700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 0x7fd64403f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.394+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd648023ec0 con 0x7fd64403cea0 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.395+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 0x7fd64403f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.395+0000 7fd65cc8f700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 0x7fd64403f290 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd654003a10 tx=0x7fd6540092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.396+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd648023ec0 con 0x7fd64403cea0 2024-04-22T06:28:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.402+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd65810cb60 con 0x7fd64403cea0 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.402+0000 7fd64e7fc700 1 -- 172.21.15.105:0/2459781207 <== mgr.14120 v2:172.21.15.105:6800/3871303088 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd65810cb60 con 0x7fd64403cea0 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 msgr2=0x7fd64403f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 0x7fd64403f290 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd654003a10 tx=0x7fd6540092b0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 msgr2=0x7fd6581091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd648003f40 tx=0x7fd648011890 comp rx=0 tx=0).stop 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 shutdown_connections 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd658109b90 0x7fd6581091b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fd64403cea0 0x7fd64403f290 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 --2- 172.21.15.105:0/2459781207 >> [v2:172.21.15.105:6800/2332170576,v1:172.21.15.105:6801/2332170576] conn(0x7fd6440395b0 0x7fd64403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 >> 172.21.15.105:0/2459781207 conn(0x7fd658102210 msgr2=0x7fd658102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 shutdown_connections 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:18.403+0000 7fd65f6f4700 1 -- 172.21.15.105:0/2459781207 wait complete. 2024-04-22T06:28:19.023 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 5 is available 2024-04-22T06:28:19.024 INFO:teuthology.orchestra.run.smithi105.stdout:Setting orchestrator backend to cephadm... 2024-04-22T06:28:19.651 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:19 smithi105 ceph-mon[27916]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:28:19.651 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:19 smithi105 ceph-mon[27916]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:28:19.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:19 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:18] ENGINE Bus STARTING 2024-04-22T06:28:19.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:19 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.409+0000 7f4643584700 1 Processor -- start 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4643584700 1 -- start start 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4643584700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4643584700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f463c108ff0 con 0x7f463c108600 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4641320700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4641320700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38124/0 (socket says 172.21.15.105:38124) 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4641320700 1 -- 172.21.15.105:0/1461048665 learned_addr learned my addr 172.21.15.105:0/1461048665 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:20.272 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.410+0000 7f4641320700 1 -- 172.21.15.105:0/1461048665 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c1097b0 con 0x7f463c108600 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4641320700 1 --2- 172.21.15.105:0/1461048665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46380098a0 tx=0x7f4638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e2b68bfef4d1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4633fff700 1 -- 172.21.15.105:0/1461048665 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4638004160 con 0x7f463c108600 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4633fff700 1 -- 172.21.15.105:0/1461048665 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4638004300 con 0x7f463c108600 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4633fff700 1 -- 172.21.15.105:0/1461048665 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4638011af0 con 0x7f463c108600 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 -- 172.21.15.105:0/1461048665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 msgr2=0x7f463c108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 --2- 172.21.15.105:0/1461048665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46380098a0 tx=0x7f4638009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:20.273 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 -- 172.21.15.105:0/1461048665 shutdown_connections 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 --2- 172.21.15.105:0/1461048665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c108a20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 -- 172.21.15.105:0/1461048665 >> 172.21.15.105:0/1461048665 conn(0x7f463c103c40 msgr2=0x7f463c106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 -- 172.21.15.105:0/1461048665 shutdown_connections 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.411+0000 7f4643584700 1 -- 172.21.15.105:0/1461048665 wait complete. 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4643584700 1 Processor -- start 2024-04-22T06:28:20.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4643584700 1 -- start start 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4643584700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4643584700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f463c10c9e0 con 0x7f463c108600 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4641320700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4641320700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38138/0 (socket says 172.21.15.105:38138) 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4641320700 1 -- 172.21.15.105:0/1956901823 learned_addr learned my addr 172.21.15.105:0/1956901823 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:20.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.412+0000 7f4641320700 1 -- 172.21.15.105:0/1956901823 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c07e0a0 con 0x7f463c108600 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f4641320700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4638009fb0 tx=0x7f4638011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46380036a0 con 0x7f463c108600 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f463801a430 con 0x7f463c108600 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c07c290 con 0x7f463c108600 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4638019550 con 0x7f463c108600 2024-04-22T06:28:20.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.413+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c07c7d0 con 0x7f463c108600 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f4638019800 con 0x7f463c108600 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c10cb60 con 0x7f463c108600 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f46327fc700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 0x7f462803b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f463804d6e0 con 0x7f463c108600 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f4640b1f700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 0x7f462803b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.414+0000 7f4640b1f700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 0x7f462803b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f462c006fd0 tx=0x7f462c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.419+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f463801a690 con 0x7f463c108600 2024-04-22T06:28:20.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.448+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4638019af0 con 0x7f463c108600 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.634+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f463c07d020 con 0x7f4628039480 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.639+0000 7f46327fc700 1 -- 172.21.15.105:0/1956901823 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f463c07d020 con 0x7f4628039480 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 msgr2=0x7f462803b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 0x7f462803b940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f462c006fd0 tx=0x7f462c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 msgr2=0x7f463c07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4638009fb0 tx=0x7f4638011940 comp rx=0 tx=0).stop 2024-04-22T06:28:20.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 shutdown_connections 2024-04-22T06:28:20.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4628039480 0x7f462803b940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:20.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 --2- 172.21.15.105:0/1956901823 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f463c108600 0x7f463c07db60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:20.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 >> 172.21.15.105:0/1956901823 conn(0x7f463c103c40 msgr2=0x7f463c104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:20.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 shutdown_connections 2024-04-22T06:28:20.279 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:19.641+0000 7f4643584700 1 -- 172.21.15.105:0/1956901823 wait complete. 2024-04-22T06:28:20.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:18] ENGINE Serving on https://172.21.15.105:7150 2024-04-22T06:28:20.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:18] ENGINE Serving on http://172.21.15.105:8765 2024-04-22T06:28:20.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:18] ENGINE Bus STARTED 2024-04-22T06:28:20.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: mgrmap e8: x(active, since 2s) 2024-04-22T06:28:20.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:20.575 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:20.575 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:20 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.674+0000 7f1c55b1a700 1 Processor -- start 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.674+0000 7f1c55b1a700 1 -- start start 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c55b1a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c55b1a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c50108f60 con 0x7f1c50108570 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38148/0 (socket says 172.21.15.105:38148) 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4f7fe700 1 -- 172.21.15.105:0/2045386033 learned_addr learned my addr 172.21.15.105:0/2045386033 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4f7fe700 1 -- 172.21.15.105:0/2045386033 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c50109770 con 0x7f1c50108570 2024-04-22T06:28:21.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4f7fe700 1 --2- 172.21.15.105:0/2045386033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1c400098a0 tx=0x7f1c40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1588e23d458e9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4e7fc700 1 -- 172.21.15.105:0/2045386033 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c40004160 con 0x7f1c50108570 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4e7fc700 1 -- 172.21.15.105:0/2045386033 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1c40004300 con 0x7f1c50108570 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.675+0000 7f1c4e7fc700 1 -- 172.21.15.105:0/2045386033 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c40011af0 con 0x7f1c50108570 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2045386033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 msgr2=0x7f1c50108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2045386033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1c400098a0 tx=0x7f1c40009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2045386033 shutdown_connections 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2045386033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c50108990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2045386033 >> 172.21.15.105:0/2045386033 conn(0x7f1c50103b90 msgr2=0x7f1c50105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2045386033 shutdown_connections 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.676+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2045386033 wait complete. 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c55b1a700 1 Processor -- start 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c55b1a700 1 -- start start 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c55b1a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:21.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c55b1a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5010cb20 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c4f7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38162/0 (socket says 172.21.15.105:38162) 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c4f7fe700 1 -- 172.21.15.105:0/2687226668 learned_addr learned my addr 172.21.15.105:0/2687226668 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.677+0000 7f1c4f7fe700 1 -- 172.21.15.105:0/2687226668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5019ca00 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c4f7fe700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1c40003f40 tx=0x7f1c40011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c400038b0 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1c40003a50 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5019cc90 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c4001a5c0 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.678+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5019f980 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.679+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1c4001a870 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.679+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c5019d170 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.679+0000 7f1c4cff9700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 0x7f1c3803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.679+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c4004e980 con 0x7f1c50108570 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.679+0000 7f1c4effd700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 0x7f1c3803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:21.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.680+0000 7f1c4effd700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 0x7f1c3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1c44006fd0 tx=0x7f1c44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.684+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c5019d170 con 0x7f1c50108570 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.888+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f1c50104d70 con 0x7f1c380395b0 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.888+0000 7f1c4cff9700 1 -- 172.21.15.105:0/2687226668 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f1c50104d70 con 0x7f1c380395b0 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 msgr2=0x7f1c3803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 0x7f1c3803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1c44006fd0 tx=0x7f1c44006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 msgr2=0x7f1c5019c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1c40003f40 tx=0x7f1c40011890 comp rx=0 tx=0).stop 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 shutdown_connections 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1c380395b0 0x7f1c3803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 --2- 172.21.15.105:0/2687226668 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c50108570 0x7f1c5019c4c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 >> 172.21.15.105:0/2687226668 conn(0x7f1c50103b90 msgr2=0x7f1c50104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 shutdown_connections 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:20.891+0000 7f1c55b1a700 1 -- 172.21.15.105:0/2687226668 wait complete. 2024-04-22T06:28:21.614 INFO:teuthology.orchestra.run.smithi105.stdout:Generating ssh key... 2024-04-22T06:28:21.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:21 smithi105 ceph-mon[27916]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:22.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: Generating public/private rsa key pair. 2024-04-22T06:28:22.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: Your identification has been saved in /tmp/tmp48gtgain/key. 2024-04-22T06:28:22.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: Your public key has been saved in /tmp/tmp48gtgain/key.pub. 2024-04-22T06:28:22.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: The key fingerprint is: 2024-04-22T06:28:22.919 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: SHA256:dwjtu/Avn3r2fPlwA9Cr69FJYp2XnEvWTB1tUjYkNqE ceph-48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: The key's randomart image is: 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: +---[RSA 3072]----+ 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | =o=+| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | . + +.*| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | . .E . oo| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | o .o +o+| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | S +o.* Bo| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | ..o= * .| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | . .o o.oo| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | o..=o +o| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: | =X+.o.o| 2024-04-22T06:28:22.920 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: +----[SHA256]-----+ 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.045+0000 7f2b527e0700 1 Processor -- start 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.045+0000 7f2b527e0700 1 -- start start 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.045+0000 7f2b527e0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.045+0000 7f2b527e0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c109450 con 0x7f2b4c108a60 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:23.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38166/0 (socket says 172.21.15.105:38166) 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4bfff700 1 -- 172.21.15.105:0/3196500708 learned_addr learned my addr 172.21.15.105:0/3196500708 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4bfff700 1 -- 172.21.15.105:0/3196500708 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c1076c0 con 0x7f2b4c108a60 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4bfff700 1 --2- 172.21.15.105:0/3196500708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0098a0 tx=0x7f2b3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc102f7e64fee849 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4affd700 1 -- 172.21.15.105:0/3196500708 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b3c004160 con 0x7f2b4c108a60 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4affd700 1 -- 172.21.15.105:0/3196500708 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2b3c004300 con 0x7f2b4c108a60 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.046+0000 7f2b4affd700 1 -- 172.21.15.105:0/3196500708 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b3c011af0 con 0x7f2b4c108a60 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- 172.21.15.105:0/3196500708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 msgr2=0x7f2b4c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3196500708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0098a0 tx=0x7f2b3c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- 172.21.15.105:0/3196500708 shutdown_connections 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3196500708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c108e80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- 172.21.15.105:0/3196500708 >> 172.21.15.105:0/3196500708 conn(0x7f2b4c102150 msgr2=0x7f2b4c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- 172.21.15.105:0/3196500708 shutdown_connections 2024-04-22T06:28:23.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- 172.21.15.105:0/3196500708 wait complete. 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 Processor -- start 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.047+0000 7f2b527e0700 1 -- start start 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b527e0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b527e0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c10ba80 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b4bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b4bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38178/0 (socket says 172.21.15.105:38178) 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b4bfff700 1 -- 172.21.15.105:0/3951355609 learned_addr learned my addr 172.21.15.105:0/3951355609 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b4bfff700 1 -- 172.21.15.105:0/3951355609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c19b9a0 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b4bfff700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0117b0 tx=0x7f2b3c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b3c003af0 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2b3c020070 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c19bc30 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b3c019630 con 0x7f2b4c108a60 2024-04-22T06:28:23.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.048+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c19e920 con 0x7f2b4c108a60 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.049+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2b3c0198a0 con 0x7f2b4c108a60 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.049+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c0640a0 con 0x7f2b4c108a60 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.049+0000 7f2b497fa700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 0x7f2b3403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.049+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b3c04dc60 con 0x7f2b4c108a60 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.049+0000 7f2b4b7fe700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 0x7f2b3403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.050+0000 7f2b4b7fe700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 0x7f2b3403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.054+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b3c00b480 con 0x7f2b4c108a60 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.251+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2b4c103430 con 0x7f2b34039600 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.666+0000 7f2b497fa700 1 -- 172.21.15.105:0/3951355609 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2b4c103430 con 0x7f2b34039600 2024-04-22T06:28:23.229 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 msgr2=0x7f2b3403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 0x7f2b3403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 msgr2=0x7f2b4c19b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0117b0 tx=0x7f2b3c011b70 comp rx=0 tx=0).stop 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 shutdown_connections 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f2b34039600 0x7f2b3403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 --2- 172.21.15.105:0/3951355609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2b4c108a60 0x7f2b4c19b460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 >> 172.21.15.105:0/3951355609 conn(0x7f2b4c102150 msgr2=0x7f2b4c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 shutdown_connections 2024-04-22T06:28:23.230 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:22.668+0000 7f2b527e0700 1 -- 172.21.15.105:0/3951355609 wait complete. 2024-04-22T06:28:23.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:23 smithi105 ceph-mon[27916]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:23.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:23 smithi105 ceph-mon[27916]: Generating ssh key... 2024-04-22T06:28:23.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:23 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:23.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:23 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:24.431 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:24.431 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d73a70700 1 Processor -- start 2024-04-22T06:28:24.431 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d73a70700 1 -- start start 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d73a70700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d73a70700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c07be20 con 0x7f4d6c106e60 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d7180c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d7180c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38190/0 (socket says 172.21.15.105:38190) 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d7180c700 1 -- 172.21.15.105:0/1139156547 learned_addr learned my addr 172.21.15.105:0/1139156547 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.626+0000 7f4d7180c700 1 -- 172.21.15.105:0/1139156547 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c1077c0 con 0x7f4d6c106e60 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d7180c700 1 --2- 172.21.15.105:0/1139156547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1988f9e47b3f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d7080a700 1 -- 172.21.15.105:0/1139156547 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68004160 con 0x7f4d6c106e60 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d7080a700 1 -- 172.21.15.105:0/1139156547 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d68004300 con 0x7f4d6c106e60 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d7080a700 1 -- 172.21.15.105:0/1139156547 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68011af0 con 0x7f4d6c106e60 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d73a70700 1 -- 172.21.15.105:0/1139156547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 msgr2=0x7f4d6c107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d73a70700 1 --2- 172.21.15.105:0/1139156547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d680098a0 tx=0x7f4d68009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:24.432 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d73a70700 1 -- 172.21.15.105:0/1139156547 shutdown_connections 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d73a70700 1 --2- 172.21.15.105:0/1139156547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c107280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.627+0000 7f4d73a70700 1 -- 172.21.15.105:0/1139156547 >> 172.21.15.105:0/1139156547 conn(0x7f4d6c102a80 msgr2=0x7f4d6c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 -- 172.21.15.105:0/1139156547 shutdown_connections 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 -- 172.21.15.105:0/1139156547 wait complete. 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 Processor -- start 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 -- start start 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d73a70700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c1087f0 con 0x7f4d6c106e60 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d7180c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d7180c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38204/0 (socket says 172.21.15.105:38204) 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.628+0000 7f4d7180c700 1 -- 172.21.15.105:0/2561700136 learned_addr learned my addr 172.21.15.105:0/2561700136 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d7180c700 1 -- 172.21.15.105:0/2561700136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c19ec40 con 0x7f4d6c106e60 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d7180c700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009fb0 tx=0x7f4d68011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:24.433 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d680036a0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d6801a430 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d68019550 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c19eed0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.629+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c1a1bc0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4d6801a8e0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6c0640a0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d62ffd700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 0x7f4d5803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4d6804d8d0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d7100b700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 0x7f4d5803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.630+0000 7f4d7100b700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 0x7f4d5803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c006fd0 tx=0x7f4d5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.635+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d6800b3e0 con 0x7f4d6c106e60 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.831+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f4d6c0008d0 con 0x7f4d580395b0 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.832+0000 7f4d62ffd700 1 -- 172.21.15.105:0/2561700136 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f4d6c0008d0 con 0x7f4d580395b0 2024-04-22T06:28:24.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 msgr2=0x7f4d5803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 0x7f4d5803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c006fd0 tx=0x7f4d5c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 msgr2=0x7f4d6c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009fb0 tx=0x7f4d68011940 comp rx=0 tx=0).stop 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 shutdown_connections 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f4d580395b0 0x7f4d5803ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 --2- 172.21.15.105:0/2561700136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d6c106e60 0x7f4d6c19e700 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 >> 172.21.15.105:0/2561700136 conn(0x7f4d6c102a80 msgr2=0x7f4d6c195370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 shutdown_connections 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:23.834+0000 7f4d73a70700 1 -- 172.21.15.105:0/2561700136 wait complete. 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:Adding key to root@localhost authorized_keys... 2024-04-22T06:28:24.435 INFO:teuthology.orchestra.run.smithi105.stdout:Adding host smithi105... 2024-04-22T06:28:24.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:24 smithi105 ceph-mon[27916]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:25.892 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:25 smithi105 ceph-mon[27916]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:25.892 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:25 smithi105 ceph-mon[27916]: Deploying cephadm binary to smithi105 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Added host 'smithi105' with addr '172.21.15.105' 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f795ddf9700 1 Processor -- start 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f795ddf9700 1 -- start start 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f795ddf9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f795ddf9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7958107230 con 0x7f7958106840 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f79577fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f79577fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50764/0 (socket says 172.21.15.105:50764) 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f79577fe700 1 -- 172.21.15.105:0/1957106125 learned_addr learned my addr 172.21.15.105:0/1957106125 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:29.632 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f79577fe700 1 -- 172.21.15.105:0/1957106125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79581097d0 con 0x7f7958106840 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.828+0000 7f79577fe700 1 --2- 172.21.15.105:0/1957106125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f79480098a0 tx=0x7f7948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f328f03c3cc8328c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f79567fc700 1 -- 172.21.15.105:0/1957106125 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948004160 con 0x7f7958106840 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f79567fc700 1 -- 172.21.15.105:0/1957106125 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7948004300 con 0x7f7958106840 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f79567fc700 1 -- 172.21.15.105:0/1957106125 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948011af0 con 0x7f7958106840 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 -- 172.21.15.105:0/1957106125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 msgr2=0x7f7958106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 --2- 172.21.15.105:0/1957106125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f79480098a0 tx=0x7f7948009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 -- 172.21.15.105:0/1957106125 shutdown_connections 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 --2- 172.21.15.105:0/1957106125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958106c60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 -- 172.21.15.105:0/1957106125 >> 172.21.15.105:0/1957106125 conn(0x7f79581021b0 msgr2=0x7f79581045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 -- 172.21.15.105:0/1957106125 shutdown_connections 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.829+0000 7f795ddf9700 1 -- 172.21.15.105:0/1957106125 wait complete. 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f795ddf9700 1 Processor -- start 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f795ddf9700 1 -- start start 2024-04-22T06:28:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f795ddf9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f795ddf9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f795810cb80 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f79577fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f79577fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50766/0 (socket says 172.21.15.105:50766) 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.830+0000 7f79577fe700 1 -- 172.21.15.105:0/3948877607 learned_addr learned my addr 172.21.15.105:0/3948877607 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f79577fe700 1 -- 172.21.15.105:0/3948877607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7958109590 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f79577fe700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7948009fb0 tx=0x7f7948011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79480036a0 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f794801a430 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7958107780 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948019550 con 0x7f7958106840 2024-04-22T06:28:29.634 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.831+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7958107cc0 con 0x7f7958106840 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7948019850 con 0x7f7958106840 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79580640a0 con 0x7f7958106840 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f7954ff9700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 0x7f794003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f794804d930 con 0x7f7958106840 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f7956ffd700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 0x7f794003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.832+0000 7f7956ffd700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 0x7f794003ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f794c006fd0 tx=0x7f794c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:24.837+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f794800fda0 con 0x7f7958106840 2024-04-22T06:28:29.635 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:25.044+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}) -- 0x7f7958108520 con 0x7f7940039560 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.092+0000 7f7954ff9700 1 -- 172.21.15.105:0/3948877607 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7958108520 con 0x7f7940039560 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 msgr2=0x7f794003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 0x7f794003ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f794c006fd0 tx=0x7f794c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 msgr2=0x7f7958109050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7948009fb0 tx=0x7f7948011940 comp rx=0 tx=0).stop 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 shutdown_connections 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7940039560 0x7f794003ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 --2- 172.21.15.105:0/3948877607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7958106840 0x7f7958109050 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 >> 172.21.15.105:0/3948877607 conn(0x7f79581021b0 msgr2=0x7f7958102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 shutdown_connections 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:29.095+0000 7f795ddf9700 1 -- 172.21.15.105:0/3948877607 wait complete. 2024-04-22T06:28:29.636 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying unmanaged mon service... 2024-04-22T06:28:30.343 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:30 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:30.343 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:30 smithi105 ceph-mon[27916]: Added host smithi105 2024-04-22T06:28:30.343 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:30 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:30.915 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-22T06:28:30.915 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.095+0000 7fc5d442f700 1 Processor -- start 2024-04-22T06:28:30.915 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.095+0000 7fc5d442f700 1 -- start start 2024-04-22T06:28:30.915 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.095+0000 7fc5d442f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.095+0000 7fc5d442f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc107e70 con 0x7fc5cc107480 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d21cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d21cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50768/0 (socket says 172.21.15.105:50768) 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d21cb700 1 -- 172.21.15.105:0/1884721019 learned_addr learned my addr 172.21.15.105:0/1884721019 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d21cb700 1 -- 172.21.15.105:0/1884721019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc108680 con 0x7fc5cc107480 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d21cb700 1 --2- 172.21.15.105:0/1884721019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800bb40 tx=0x7fc5c800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c95a13db4c57006 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d11c9700 1 -- 172.21.15.105:0/1884721019 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5c800dbe0 con 0x7fc5cc107480 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d11c9700 1 -- 172.21.15.105:0/1884721019 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc5c80044c0 con 0x7fc5cc107480 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.096+0000 7fc5d11c9700 1 -- 172.21.15.105:0/1884721019 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5c8010470 con 0x7fc5cc107480 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- 172.21.15.105:0/1884721019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 msgr2=0x7fc5cc1078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1884721019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800bb40 tx=0x7fc5c800be50 comp rx=0 tx=0).stop 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- 172.21.15.105:0/1884721019 shutdown_connections 2024-04-22T06:28:30.916 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1884721019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc1078a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- 172.21.15.105:0/1884721019 >> 172.21.15.105:0/1884721019 conn(0x7fc5cc102a80 msgr2=0x7fc5cc104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- 172.21.15.105:0/1884721019 shutdown_connections 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- 172.21.15.105:0/1884721019 wait complete. 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 Processor -- start 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.097+0000 7fc5d442f700 1 -- start start 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d442f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d442f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc062e80 con 0x7fc5cc107480 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d21cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d21cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50784/0 (socket says 172.21.15.105:50784) 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d21cb700 1 -- 172.21.15.105:0/1164860364 learned_addr learned my addr 172.21.15.105:0/1164860364 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d21cb700 1 -- 172.21.15.105:0/1164860364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc19a780 con 0x7fc5cc107480 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d21cb700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800b340 tx=0x7fc5c8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5c8027070 con 0x7fc5cc107480 2024-04-22T06:28:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc5c8010bb0 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc19aa10 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5c8022ac0 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.098+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc19d700 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.099+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5cc0640a0 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.099+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc5c802a070 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.099+0000 7fc5c37fe700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 0x7fc5b803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.099+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc5c801f070 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.100+0000 7fc5d19ca700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 0x7fc5b803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.100+0000 7fc5d19ca700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 0x7fc5b803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.104+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5c8015070 con 0x7fc5cc107480 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.320+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc5cc075460 con 0x7fc5b80395b0 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.329+0000 7fc5c37fe700 1 -- 172.21.15.105:0/1164860364 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc5cc075460 con 0x7fc5b80395b0 2024-04-22T06:28:30.918 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 msgr2=0x7fc5b803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 0x7fc5b803ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 msgr2=0x7fc5cc19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5c800b340 tx=0x7fc5c8003dc0 comp rx=0 tx=0).stop 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 shutdown_connections 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fc5b80395b0 0x7fc5b803ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 --2- 172.21.15.105:0/1164860364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5cc107480 0x7fc5cc19a240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 >> 172.21.15.105:0/1164860364 conn(0x7fc5cc102a80 msgr2=0x7fc5cc190860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 shutdown_connections 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:30.331+0000 7fc5d442f700 1 -- 172.21.15.105:0/1164860364 wait complete. 2024-04-22T06:28:30.919 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying unmanaged mgr service... 2024-04-22T06:28:31.479 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:31 smithi105 ceph-mon[27916]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:31.479 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:31 smithi105 ceph-mon[27916]: Saving service mon spec with placement count:5 2024-04-22T06:28:31.479 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:31 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:32.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-22T06:28:32.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7be4b56700 1 Processor -- start 2024-04-22T06:28:32.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7be4b56700 1 -- start start 2024-04-22T06:28:32.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7be4b56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:32.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7be4b56700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7be0107e50 con 0x7f7be0107460 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7bde59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7bde59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50790/0 (socket says 172.21.15.105:50790) 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.309+0000 7f7bde59c700 1 -- 172.21.15.105:0/4120643076 learned_addr learned my addr 172.21.15.105:0/4120643076 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.310+0000 7f7bde59c700 1 -- 172.21.15.105:0/4120643076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be0108660 con 0x7f7be0107460 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.310+0000 7f7bde59c700 1 --2- 172.21.15.105:0/4120643076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00098a0 tx=0x7f7bd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c43badb74660ec server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.310+0000 7f7bdd59a700 1 -- 172.21.15.105:0/4120643076 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd0004160 con 0x7f7be0107460 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.310+0000 7f7bdd59a700 1 -- 172.21.15.105:0/4120643076 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7bd0004300 con 0x7f7be0107460 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.310+0000 7f7bdd59a700 1 -- 172.21.15.105:0/4120643076 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd0011af0 con 0x7f7be0107460 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 -- 172.21.15.105:0/4120643076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 msgr2=0x7f7be0107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 --2- 172.21.15.105:0/4120643076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00098a0 tx=0x7f7bd0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:32.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 -- 172.21.15.105:0/4120643076 shutdown_connections 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 --2- 172.21.15.105:0/4120643076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be0107880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 -- 172.21.15.105:0/4120643076 >> 172.21.15.105:0/4120643076 conn(0x7f7be0102a40 msgr2=0x7f7be0104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 -- 172.21.15.105:0/4120643076 shutdown_connections 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 -- 172.21.15.105:0/4120643076 wait complete. 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.311+0000 7f7be4b56700 1 Processor -- start 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7be4b56700 1 -- start start 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7be4b56700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7be4b56700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7be010ba10 con 0x7f7be0107460 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bde59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bde59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50806/0 (socket says 172.21.15.105:50806) 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bde59c700 1 -- 172.21.15.105:0/940227318 learned_addr learned my addr 172.21.15.105:0/940227318 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bde59c700 1 -- 172.21.15.105:0/940227318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be019b9a0 con 0x7f7be0107460 2024-04-22T06:28:32.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bde59c700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00117b0 tx=0x7f7bd0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd0003af0 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7bd0020070 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bd0019630 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be019bc30 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.312+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be019e920 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.313+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7bd00198f0 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.313+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be00640a0 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.313+0000 7f7bcf7fe700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 0x7f7bc803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.314+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7bd004db80 con 0x7f7be0107460 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.314+0000 7f7bddd9b700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 0x7f7bc803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:32.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.314+0000 7f7bddd9b700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 0x7f7bc803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006fd0 tx=0x7f7bd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.318+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bd000e260 con 0x7f7be0107460 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.535+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f7be0103d50 con 0x7f7bc80395b0 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.542+0000 7f7bcf7fe700 1 -- 172.21.15.105:0/940227318 <== mgr.14120 v2:172.21.15.105:6800/3871303088 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7be0103d50 con 0x7f7bc80395b0 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 msgr2=0x7f7bc803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 0x7f7bc803ba70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4006fd0 tx=0x7f7bd4006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 msgr2=0x7f7be019b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00117b0 tx=0x7f7bd0011b70 comp rx=0 tx=0).stop 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 shutdown_connections 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7bc80395b0 0x7f7bc803ba70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 --2- 172.21.15.105:0/940227318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be0107460 0x7f7be019b460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 >> 172.21.15.105:0/940227318 conn(0x7f7be0102a40 msgr2=0x7f7be0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:32.179 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 shutdown_connections 2024-04-22T06:28:32.180 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:31.544+0000 7f7be4b56700 1 -- 172.21.15.105:0/940227318 wait complete. 2024-04-22T06:28:32.895 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:32 smithi105 ceph-mon[27916]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:32.895 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:32 smithi105 ceph-mon[27916]: Saving service mgr spec with placement count:2 2024-04-22T06:28:32.895 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:32 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:34.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:33 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:34.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:33 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:34.161 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:33 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2758322136' entity='client.admin' 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f96e59700 1 Processor -- start 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f96e59700 1 -- start start 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f96e59700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f96e59700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9007be60 con 0x7f7f90109030 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f94bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f94bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50810/0 (socket says 172.21.15.105:50810) 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f94bf5700 1 -- 172.21.15.105:0/3427249395 learned_addr learned my addr 172.21.15.105:0/3427249395 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.352+0000 7f7f94bf5700 1 -- 172.21.15.105:0/3427249395 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9010b960 con 0x7f7f90109030 2024-04-22T06:28:34.167 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f94bf5700 1 --2- 172.21.15.105:0/3427249395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f840098a0 tx=0x7f7f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881bf4ca265a778c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f8f7fe700 1 -- 172.21.15.105:0/3427249395 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f84004160 con 0x7f7f90109030 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f8f7fe700 1 -- 172.21.15.105:0/3427249395 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f84004300 con 0x7f7f90109030 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f8f7fe700 1 -- 172.21.15.105:0/3427249395 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f84011a20 con 0x7f7f90109030 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f96e59700 1 -- 172.21.15.105:0/3427249395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 msgr2=0x7f7f9010b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f96e59700 1 --2- 172.21.15.105:0/3427249395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f840098a0 tx=0x7f7f84009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f96e59700 1 -- 172.21.15.105:0/3427249395 shutdown_connections 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f96e59700 1 --2- 172.21.15.105:0/3427249395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f9010b420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.353+0000 7f7f96e59700 1 -- 172.21.15.105:0/3427249395 >> 172.21.15.105:0/3427249395 conn(0x7f7f90102a80 msgr2=0x7f7f90104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 -- 172.21.15.105:0/3427249395 shutdown_connections 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 -- 172.21.15.105:0/3427249395 wait complete. 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 Processor -- start 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 -- start start 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:34.168 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f96e59700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9010c960 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f94bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f94bf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50826/0 (socket says 172.21.15.105:50826) 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.354+0000 7f7f94bf5700 1 -- 172.21.15.105:0/2758322136 learned_addr learned my addr 172.21.15.105:0/2758322136 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f94bf5700 1 -- 172.21.15.105:0/2758322136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f90108df0 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f94bf5700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f840116e0 tx=0x7f7f84011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f84003b20 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f90106fe0 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f8401b8f0 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7f840238c0 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.355+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f90107520 con 0x7f7f90109030 2024-04-22T06:28:34.169 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.356+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7f8401baa0 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.356+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f900640a0 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.356+0000 7f7f8dffb700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 0x7f7f8003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.356+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7f8404e060 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.357+0000 7f7f8ffff700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 0x7f7f8003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.357+0000 7f7f8ffff700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 0x7f7f8003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c006fd0 tx=0x7f7f7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.361+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f8401f030 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.593+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7f90107dc0 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.598+0000 7f7f8dffb700 1 -- 172.21.15.105:0/2758322136 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7f840292a0 con 0x7f7f90109030 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 msgr2=0x7f7f8003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 0x7f7f8003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c006fd0 tx=0x7f7f7c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 msgr2=0x7f7f901088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7f840116e0 tx=0x7f7f84011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 shutdown_connections 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f7f80039560 0x7f7f8003ba20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 --2- 172.21.15.105:0/2758322136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f90109030 0x7f7f901088b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 >> 172.21.15.105:0/2758322136 conn(0x7f7f90102a80 msgr2=0x7f7f901035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 shutdown_connections 2024-04-22T06:28:34.171 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:33.601+0000 7f7f96e59700 1 -- 172.21.15.105:0/2758322136 wait complete. 2024-04-22T06:28:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:35 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/755254528' entity='client.admin' 2024-04-22T06:28:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:35 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26d5deb700 1 Processor -- start 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26d5deb700 1 -- start start 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26d5deb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26d5deb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26d007c8e0 con 0x7f26d007de60 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54584/0 (socket says 172.21.15.105:54584) 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.605+0000 7f26cf7fe700 1 -- 172.21.15.105:0/3103269770 learned_addr learned my addr 172.21.15.105:0/3103269770 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26cf7fe700 1 -- 172.21.15.105:0/3103269770 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d007ca60 con 0x7f26d007de60 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26cf7fe700 1 --2- 172.21.15.105:0/3103269770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26c00098a0 tx=0x7f26c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc739eb5a3a6d7b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:35.960 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26ceffd700 1 -- 172.21.15.105:0/3103269770 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c0004160 con 0x7f26d007de60 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26ceffd700 1 -- 172.21.15.105:0/3103269770 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26c0004300 con 0x7f26d007de60 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26ceffd700 1 -- 172.21.15.105:0/3103269770 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c0011a20 con 0x7f26d007de60 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26d5deb700 1 -- 172.21.15.105:0/3103269770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 msgr2=0x7f26d007c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26d5deb700 1 --2- 172.21.15.105:0/3103269770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26c00098a0 tx=0x7f26c0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26d5deb700 1 -- 172.21.15.105:0/3103269770 shutdown_connections 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26d5deb700 1 --2- 172.21.15.105:0/3103269770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d007c310 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.606+0000 7f26d5deb700 1 -- 172.21.15.105:0/3103269770 >> 172.21.15.105:0/3103269770 conn(0x7f26d0103c40 msgr2=0x7f26d0106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 -- 172.21.15.105:0/3103269770 shutdown_connections 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 -- 172.21.15.105:0/3103269770 wait complete. 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 Processor -- start 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 -- start start 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26d5deb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26d010eb80 con 0x7f26d007de60 2024-04-22T06:28:35.961 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26cf7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54588/0 (socket says 172.21.15.105:54588) 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.607+0000 7f26cf7fe700 1 -- 172.21.15.105:0/755254528 learned_addr learned my addr 172.21.15.105:0/755254528 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26cf7fe700 1 -- 172.21.15.105:0/755254528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d019ebc0 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26cf7fe700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26c0004660 tx=0x7f26c0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c0003aa0 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26c001b8f0 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d019ee50 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c0023900 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.608+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d01a1b40 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.609+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f26c0023b70 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.609+0000 7f26cd7fa700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 0x7f26bc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.609+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f26c004edf0 con 0x7f26d007de60 2024-04-22T06:28:35.962 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.609+0000 7f26c6ffd700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 0x7f26bc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.609+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d00640a0 con 0x7f26d007de60 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.610+0000 7f26c6ffd700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 0x7f26bc03bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f26b8006fd0 tx=0x7f26b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.614+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26c001f030 con 0x7f26d007de60 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.837+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f26d01a1d10 con 0x7f26d007de60 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.839+0000 7f26cd7fa700 1 -- 172.21.15.105:0/755254528 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f26c0018e30 con 0x7f26d007de60 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 msgr2=0x7f26bc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 0x7f26bc03bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f26b8006fd0 tx=0x7f26b8006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 msgr2=0x7f26d019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f26c0004660 tx=0x7f26c0011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 shutdown_connections 2024-04-22T06:28:35.963 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f26bc039600 0x7f26bc03bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:35.964 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 --2- 172.21.15.105:0/755254528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26d007de60 0x7f26d019e680 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:35.964 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 >> 172.21.15.105:0/755254528 conn(0x7f26d0103c40 msgr2=0x7f26d0104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:35.964 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 shutdown_connections 2024-04-22T06:28:35.964 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:34.842+0000 7f26d5deb700 1 -- 172.21.15.105:0/755254528 wait complete. 2024-04-22T06:28:35.964 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling the dashboard module... 2024-04-22T06:28:37.822 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:37 smithi105 ceph-mon[27916]: from='mgr.14120 172.21.15.105:0/732070430' entity='mgr.x' 2024-04-22T06:28:37.822 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:37 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2763826073' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T06:28:37.822 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setuser ceph since I am not root 2024-04-22T06:28:37.822 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setgroup ceph since I am not root 2024-04-22T06:28:37.822 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:37.822+0000 7fdaf673c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:28:38.111 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:38.009+0000 7fdaf673c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:28:38.111 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:38.112+0000 7fdaf673c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.368+0000 7fb4a40a8700 1 Processor -- start 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.368+0000 7fb4a40a8700 1 -- start start 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.368+0000 7fb4a40a8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.368+0000 7fb4a40a8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c063d00 con 0x7fb49c109050 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a1e44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a1e44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54592/0 (socket says 172.21.15.105:54592) 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a1e44700 1 -- 172.21.15.105:0/1797154780 learned_addr learned my addr 172.21.15.105:0/1797154780 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a1e44700 1 -- 172.21.15.105:0/1797154780 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c10b980 con 0x7fb49c109050 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a1e44700 1 --2- 172.21.15.105:0/1797154780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb4980098a0 tx=0x7fb498009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e85be9383305b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:38.246 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a0e42700 1 -- 172.21.15.105:0/1797154780 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb498004160 con 0x7fb49c109050 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a0e42700 1 -- 172.21.15.105:0/1797154780 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb498004300 con 0x7fb49c109050 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.369+0000 7fb4a0e42700 1 -- 172.21.15.105:0/1797154780 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb498011a20 con 0x7fb49c109050 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 -- 172.21.15.105:0/1797154780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 msgr2=0x7fb49c10b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/1797154780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb4980098a0 tx=0x7fb498009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 -- 172.21.15.105:0/1797154780 shutdown_connections 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/1797154780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c10b440 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 -- 172.21.15.105:0/1797154780 >> 172.21.15.105:0/1797154780 conn(0x7fb49c102a80 msgr2=0x7fb49c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 -- 172.21.15.105:0/1797154780 shutdown_connections 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.370+0000 7fb4a40a8700 1 -- 172.21.15.105:0/1797154780 wait complete. 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a40a8700 1 Processor -- start 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a40a8700 1 -- start start 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a40a8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a40a8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c10c930 con 0x7fb49c109050 2024-04-22T06:28:38.247 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a1e44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a1e44700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54602/0 (socket says 172.21.15.105:54602) 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a1e44700 1 -- 172.21.15.105:0/2763826073 learned_addr learned my addr 172.21.15.105:0/2763826073 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.371+0000 7fb4a1e44700 1 -- 172.21.15.105:0/2763826073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c19c960 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb4a1e44700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb498000c00 tx=0x7fb498011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb498003aa0 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49801b8f0 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49c19cbf0 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb49c19f8e0 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.372+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb498023870 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb49c0640a0 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb498023b20 con 0x7fb49c109050 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb492ffd700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 0x7fb48803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:38.248 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb49804de20 con 0x7fb49c109050 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb4a1643700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 0x7fb48803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.373+0000 7fb4a1643700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 0x7fb48803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.378+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb49801f030 con 0x7fb49c109050 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:36.652+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb49c19fed0 con 0x7fb49c109050 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.646+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fb49801baa0 con 0x7fb49c109050 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.647+0000 7fb492ffd700 1 -- 172.21.15.105:0/2763826073 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fb49802c6d0 con 0x7fb49c109050 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 msgr2=0x7fb48803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 0x7fb48803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 msgr2=0x7fb49c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb498000c00 tx=0x7fb498011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 shutdown_connections 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7fb488039600 0x7fb48803bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 --2- 172.21.15.105:0/2763826073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb49c109050 0x7fb49c19c420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:38.249 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 >> 172.21.15.105:0/2763826073 conn(0x7fb49c102a80 msgr2=0x7fb49c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:38.250 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 shutdown_connections 2024-04-22T06:28:38.250 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:37.649+0000 7fb4a40a8700 1 -- 172.21.15.105:0/2763826073 wait complete. 2024-04-22T06:28:38.634 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:38.426+0000 7fdaf673c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:28:38.634 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:38.515+0000 7fdaf673c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:28:38.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2763826073' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T06:28:38.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:38 smithi105 ceph-mon[27916]: mgrmap e9: x(active, since 20s) 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-22T06:28:39.537 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.634+0000 7f478f650700 1 Processor -- start 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.634+0000 7f478f650700 1 -- start start 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.634+0000 7f478f650700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.634+0000 7f478f650700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4788063d60 con 0x7f4788106de0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.634+0000 7f478d3ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f478d3ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54622/0 (socket says 172.21.15.105:54622) 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f478d3ec700 1 -- 172.21.15.105:0/588957795 learned_addr learned my addr 172.21.15.105:0/588957795 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f478d3ec700 1 -- 172.21.15.105:0/588957795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4788109750 con 0x7f4788106de0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f478d3ec700 1 --2- 172.21.15.105:0/588957795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f47840098a0 tx=0x7f4784009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fead01c2a5ed22bd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f477ffff700 1 -- 172.21.15.105:0/588957795 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4784004160 con 0x7f4788106de0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f477ffff700 1 -- 172.21.15.105:0/588957795 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4784004300 con 0x7f4788106de0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.635+0000 7f477ffff700 1 -- 172.21.15.105:0/588957795 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4784011a20 con 0x7f4788106de0 2024-04-22T06:28:39.538 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 -- 172.21.15.105:0/588957795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 msgr2=0x7f4788109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 --2- 172.21.15.105:0/588957795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f47840098a0 tx=0x7f4784009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 -- 172.21.15.105:0/588957795 shutdown_connections 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 --2- 172.21.15.105:0/588957795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f4788109210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 -- 172.21.15.105:0/588957795 >> 172.21.15.105:0/588957795 conn(0x7f4788102a80 msgr2=0x7f4788104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 -- 172.21.15.105:0/588957795 shutdown_connections 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.636+0000 7f478f650700 1 -- 172.21.15.105:0/588957795 wait complete. 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.637+0000 7f478f650700 1 Processor -- start 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.637+0000 7f478f650700 1 -- start start 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.637+0000 7f478f650700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.637+0000 7f478f650700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f478810a7c0 con 0x7f4788106de0 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478d3ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478d3ec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54626/0 (socket says 172.21.15.105:54626) 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478d3ec700 1 -- 172.21.15.105:0/3998067926 learned_addr learned my addr 172.21.15.105:0/3998067926 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:39.539 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478d3ec700 1 -- 172.21.15.105:0/3998067926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47881a2f70 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478d3ec700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47840116e0 tx=0x7f4784011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4784003a60 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f478401b8f0 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.638+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47881a3200 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.639+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4784019860 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.639+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47881a5ef0 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47880640a0 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f4784019b60 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f477e7fc700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 0x7f477403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4784053060 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f478cbeb700 1 -- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 msgr2=0x7f477403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.640+0000 7f478cbeb700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 0x7f477403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.649+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f478401f030 con 0x7f4788106de0 2024-04-22T06:28:39.540 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.841+0000 7f478cbeb700 1 -- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 msgr2=0x7f477403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.841+0000 7f478cbeb700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 0x7f477403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.933+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f478802e280 con 0x7f4788106de0 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.934+0000 7f477e7fc700 1 -- 172.21.15.105:0/3998067926 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f478401f210 con 0x7f4788106de0 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.937+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 msgr2=0x7f477403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 0x7f477403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 msgr2=0x7f47881a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47840116e0 tx=0x7f4784011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 shutdown_connections 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f47740395b0 0x7f477403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 --2- 172.21.15.105:0/3998067926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4788106de0 0x7f47881a2a30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 >> 172.21.15.105:0/3998067926 conn(0x7f4788102a80 msgr2=0x7f4788104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 shutdown_connections 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:38.938+0000 7f478f650700 1 -- 172.21.15.105:0/3998067926 wait complete. 2024-04-22T06:28:39.541 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-04-22T06:28:39.542 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 9... 2024-04-22T06:28:39.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3998067926' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T06:28:39.857 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:39.739+0000 7fdaf673c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:28:40.163 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:39.915+0000 7fdaf673c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:40.163 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:40.087+0000 7fdaf673c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:28:40.418 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:40.378+0000 7fdaf673c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:28:40.869 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:40.470+0000 7fdaf673c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:28:40.869 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:40.675+0000 7fdaf673c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:28:41.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:40.870+0000 7fdaf673c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:28:41.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:41.478+0000 7fdaf673c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:28:42.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:41.903+0000 7fdaf673c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:28:42.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:42.000+0000 7fdaf673c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:28:42.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:42.202+0000 7fdaf673c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:28:43.064 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:42.725+0000 7fdaf673c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:28:43.064 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:42.819+0000 7fdaf673c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:28:43.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:43.064+0000 7fdaf673c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:28:43.410 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:43.198+0000 7fdaf673c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:28:43.410 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:43.294+0000 7fdaf673c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:28:43.410 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:43.410+0000 7fdaf673c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:28:44.168 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:43.714+0000 7fdaf673c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:28:44.589 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:44.316+0000 7fdaf673c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:28:44.589 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:28:44.537+0000 7fdaf673c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: Active manager daemon x restarted 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: Activating manager daemon x 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: mgrmap e10: x(active, starting, since 0.0034248s) 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: Manager daemon x is now available 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:28:45.891 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:28:45.891 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:45 smithi105 ceph-mon[27916]: mgrmap e11: x(active, since 1.00663s) 2024-04-22T06:28:46.406 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.958+0000 7f1c06971700 1 Processor -- start 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.958+0000 7f1c06971700 1 -- start start 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1c06971700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1c06971700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c00107240 con 0x7f1c00106850 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54634/0 (socket says 172.21.15.105:54634) 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfffff700 1 -- 172.21.15.105:0/4186480118 learned_addr learned my addr 172.21.15.105:0/4186480118 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:46.407 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfffff700 1 -- 172.21.15.105:0/4186480118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c001073c0 con 0x7f1c00106850 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfffff700 1 --2- 172.21.15.105:0/4186480118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00098a0 tx=0x7f1bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33978e0d778175a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfeffd700 1 -- 172.21.15.105:0/4186480118 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bf0004160 con 0x7f1c00106850 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfeffd700 1 -- 172.21.15.105:0/4186480118 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf0004300 con 0x7f1c00106850 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.959+0000 7f1bfeffd700 1 -- 172.21.15.105:0/4186480118 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bf0011a20 con 0x7f1c00106850 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 -- 172.21.15.105:0/4186480118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 msgr2=0x7f1c00106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 --2- 172.21.15.105:0/4186480118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00098a0 tx=0x7f1bf0009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 -- 172.21.15.105:0/4186480118 shutdown_connections 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 --2- 172.21.15.105:0/4186480118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c00106c70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 -- 172.21.15.105:0/4186480118 >> 172.21.15.105:0/4186480118 conn(0x7f1c00102210 msgr2=0x7f1c00104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 -- 172.21.15.105:0/4186480118 shutdown_connections 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.960+0000 7f1c06971700 1 -- 172.21.15.105:0/4186480118 wait complete. 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1c06971700 1 Processor -- start 2024-04-22T06:28:46.408 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1c06971700 1 -- start start 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1c06971700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1c06971700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0010cb00 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1bfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1bfffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54636/0 (socket says 172.21.15.105:54636) 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.961+0000 7f1bfffff700 1 -- 172.21.15.105:0/2053689414 learned_addr learned my addr 172.21.15.105:0/2053689414 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1bfffff700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0019c9a0 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1bfffff700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009520 tx=0x7f1bf0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bf00039e0 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bf001b8f0 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0019cc30 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1bf0019770 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.962+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0019f920 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f1bf0019a20 con 0x7f1c00106850 2024-04-22T06:28:46.409 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bfd7fa700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1be803c220 con 0x7f1be8039650 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1bf004def0 con 0x7f1c00106850 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bff7fe700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:39.963+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:40.164+0000 7f1bff7fe700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:40.164+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:40.565+0000 7f1bff7fe700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:40.565+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:41.366+0000 7f1bff7fe700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:41.366+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:42.967+0000 7f1bff7fe700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3871303088 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:42.967+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:44.540+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f1bf001a8f0 con 0x7f1c00106850 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:44.540+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 msgr2=0x7f1be803bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:44.540+0000 7f1bfd7fa700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.542+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f1bf001a3d0 con 0x7f1c00106850 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.542+0000 7f1bfd7fa700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 0x7f1be803f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.542+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1bf000bca0 con 0x7f1be803cfe0 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.543+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 0x7f1be803f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.544+0000 7f1bff7fe700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 0x7f1be803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4003a10 tx=0x7f1bf4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.544+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f1bf000bca0 con 0x7f1be803cfe0 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.551+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1c00069bb0 con 0x7f1be803cfe0 2024-04-22T06:28:46.411 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.551+0000 7f1bfd7fa700 1 -- 172.21.15.105:0/2053689414 <== mgr.14152 v2:172.21.15.105:6800/2556874834 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f1c00069bb0 con 0x7f1be803cfe0 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 msgr2=0x7f1be803f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 0x7f1be803f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4003a10 tx=0x7f1bf4009270 comp rx=0 tx=0).stop 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 msgr2=0x7f1c0019c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009520 tx=0x7f1bf0011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 shutdown_connections 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1be803cfe0 0x7f1be803f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c00106850 0x7f1c0019c460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 --2- 172.21.15.105:0/2053689414 >> [v2:172.21.15.105:6800/3871303088,v1:172.21.15.105:6801/3871303088] conn(0x7f1be8039650 0x7f1be803bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 >> 172.21.15.105:0/2053689414 conn(0x7f1c00102210 msgr2=0x7f1c00102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 shutdown_connections 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:45.552+0000 7f1c06971700 1 -- 172.21.15.105:0/2053689414 wait complete. 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 9 is available 2024-04-22T06:28:46.412 INFO:teuthology.orchestra.run.smithi105.stdout:Generating a dashboard self-signed certificate... 2024-04-22T06:28:46.799 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:44] ENGINE Bus STARTING 2024-04-22T06:28:46.799 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:44] ENGINE Serving on https://172.21.15.105:7150 2024-04-22T06:28:46.799 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:45] ENGINE Serving on http://172.21.15.105:8765 2024-04-22T06:28:46.800 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: [22/Apr/2024:06:28:45] ENGINE Bus STARTED 2024-04-22T06:28:46.800 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T06:28:46.800 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T06:28:46.800 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f807194c700 1 Processor -- start 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f807194c700 1 -- start start 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f807194c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f807194c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f806c063ce0 con 0x7f806c106e60 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f806affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f806affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40510/0 (socket says 172.21.15.105:40510) 2024-04-22T06:28:47.928 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.844+0000 7f806affd700 1 -- 172.21.15.105:0/3819016486 learned_addr learned my addr 172.21.15.105:0/3819016486 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.845+0000 7f806affd700 1 -- 172.21.15.105:0/3819016486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c1077c0 con 0x7f806c106e60 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.845+0000 7f806affd700 1 --2- 172.21.15.105:0/3819016486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f80540098a0 tx=0x7f8054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3569dc6f1c673f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.845+0000 7f8069ffb700 1 -- 172.21.15.105:0/3819016486 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8054004160 con 0x7f806c106e60 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.845+0000 7f8069ffb700 1 -- 172.21.15.105:0/3819016486 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8054004300 con 0x7f806c106e60 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.845+0000 7f8069ffb700 1 -- 172.21.15.105:0/3819016486 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8054011a20 con 0x7f806c106e60 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- 172.21.15.105:0/3819016486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 msgr2=0x7f806c107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 --2- 172.21.15.105:0/3819016486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f80540098a0 tx=0x7f8054009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- 172.21.15.105:0/3819016486 shutdown_connections 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 --2- 172.21.15.105:0/3819016486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c107280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- 172.21.15.105:0/3819016486 >> 172.21.15.105:0/3819016486 conn(0x7f806c102a80 msgr2=0x7f806c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- 172.21.15.105:0/3819016486 shutdown_connections 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- 172.21.15.105:0/3819016486 wait complete. 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 Processor -- start 2024-04-22T06:28:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.846+0000 7f807194c700 1 -- start start 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807194c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807194c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f806c108830 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f806affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f806affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40514/0 (socket says 172.21.15.105:40514) 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f806affd700 1 -- 172.21.15.105:0/2466126359 learned_addr learned my addr 172.21.15.105:0/2466126359 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f806affd700 1 -- 172.21.15.105:0/2466126359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c19eb50 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f806affd700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80540116e0 tx=0x7f8054011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8054003a60 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f805401b8f0 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8054019860 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c19ede0 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.847+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1a1ad0 con 0x7f806c106e60 2024-04-22T06:28:47.930 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.848+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f8054019b60 con 0x7f806c106e60 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.848+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c0640a0 con 0x7f806c106e60 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.848+0000 7f807094a700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 0x7f805803b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.849+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f805404dc10 con 0x7f806c106e60 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.849+0000 7f806a7fc700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 0x7f805803b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.849+0000 7f806a7fc700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 0x7f805803b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f805c006fd0 tx=0x7f805c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.854+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f805401f030 con 0x7f806c106e60 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:46.900+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f805401a8f0 con 0x7f806c106e60 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.075+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f806c02e280 con 0x7f80580394a0 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.188+0000 7f807094a700 1 -- 172.21.15.105:0/2466126359 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f806c02e280 con 0x7f80580394a0 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 msgr2=0x7f805803b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 0x7f805803b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f805c006fd0 tx=0x7f805c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 msgr2=0x7f806c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:47.931 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f80540116e0 tx=0x7f8054011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 shutdown_connections 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f80580394a0 0x7f805803b960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 --2- 172.21.15.105:0/2466126359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f806c106e60 0x7f806c19e610 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.190+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 >> 172.21.15.105:0/2466126359 conn(0x7f806c102a80 msgr2=0x7f806c195280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.191+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 shutdown_connections 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:47.191+0000 7f807194c700 1 -- 172.21.15.105:0/2466126359 wait complete. 2024-04-22T06:28:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial admin user... 2024-04-22T06:28:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:47 smithi105 ceph-mon[27916]: mgrmap e12: x(active, since 2s) 2024-04-22T06:28:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:47 smithi105 ceph-mon[27916]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:48 smithi105 ceph-mon[27916]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:28:49.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$bs99QjGe8OlHL68FxkxoSeo5gqk.7Tkza524nweX8bpFHL0kpepSi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713767328, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc851a0700 1 Processor -- start 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc851a0700 1 -- start start 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc851a0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc851a0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc80107e50 con 0x7fdc80107460 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40526/0 (socket says 172.21.15.105:40526) 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc7ed9d700 1 -- 172.21.15.105:0/3452799816 learned_addr learned my addr 172.21.15.105:0/3452799816 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:50.111 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc7ed9d700 1 -- 172.21.15.105:0/3452799816 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc80108660 con 0x7fdc80107460 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.326+0000 7fdc7ed9d700 1 --2- 172.21.15.105:0/3452799816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc740098a0 tx=0x7fdc74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36221f779bf7f9de server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc7dd9b700 1 -- 172.21.15.105:0/3452799816 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc74004160 con 0x7fdc80107460 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc7dd9b700 1 -- 172.21.15.105:0/3452799816 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc74004300 con 0x7fdc80107460 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc7dd9b700 1 -- 172.21.15.105:0/3452799816 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc74011a20 con 0x7fdc80107460 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 -- 172.21.15.105:0/3452799816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 msgr2=0x7fdc80107880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 --2- 172.21.15.105:0/3452799816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc740098a0 tx=0x7fdc74009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 -- 172.21.15.105:0/3452799816 shutdown_connections 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 --2- 172.21.15.105:0/3452799816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc80107880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 -- 172.21.15.105:0/3452799816 >> 172.21.15.105:0/3452799816 conn(0x7fdc80102a40 msgr2=0x7fdc80104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 -- 172.21.15.105:0/3452799816 shutdown_connections 2024-04-22T06:28:50.112 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.327+0000 7fdc851a0700 1 -- 172.21.15.105:0/3452799816 wait complete. 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc851a0700 1 Processor -- start 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc851a0700 1 -- start start 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc851a0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc851a0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8010ba10 con 0x7fdc80107460 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:50.113 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc7ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40532/0 (socket says 172.21.15.105:40532) 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.328+0000 7fdc7ed9d700 1 -- 172.21.15.105:0/760093880 learned_addr learned my addr 172.21.15.105:0/760093880 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc7ed9d700 1 -- 172.21.15.105:0/760093880 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc80192b10 con 0x7fdc80107460 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc7ed9d700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc740119f0 tx=0x7fdc74011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc740039e0 con 0x7fdc80107460 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc80190d00 con 0x7fdc80107460 2024-04-22T06:28:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc7401b8f0 con 0x7fdc80107460 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc740237a0 con 0x7fdc80107460 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.329+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc80191240 con 0x7fdc80107460 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fdc74023aa0 con 0x7fdc80107460 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc67fff700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 0x7fdc6803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdc7404dfc0 con 0x7fdc80107460 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc7e59c700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 0x7fdc6803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:50.115 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc800640a0 con 0x7fdc80107460 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.330+0000 7fdc7e59c700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 0x7fdc6803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc70006fd0 tx=0x7fdc70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.335+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc7401f030 con 0x7fdc80107460 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.559+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fdc80196320 con 0x7fdc680395b0 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.783+0000 7fdc67fff700 1 -- 172.21.15.105:0/760093880 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fdc80196320 con 0x7fdc680395b0 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 msgr2=0x7fdc6803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:50.116 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 0x7fdc6803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc70006fd0 tx=0x7fdc70006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 msgr2=0x7fdc801925d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc740119f0 tx=0x7fdc74011ad0 comp rx=0 tx=0).stop 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 shutdown_connections 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdc680395b0 0x7fdc6803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 --2- 172.21.15.105:0/760093880 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc80107460 0x7fdc801925d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:50.117 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 >> 172.21.15.105:0/760093880 conn(0x7fdc80102a40 msgr2=0x7fdc801038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:50.118 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 shutdown_connections 2024-04-22T06:28:50.118 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:48.785+0000 7fdc851a0700 1 -- 172.21.15.105:0/760093880 wait complete. 2024-04-22T06:28:50.118 INFO:teuthology.orchestra.run.smithi105.stdout:Fetching dashboard port number... 2024-04-22T06:28:51.332 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 8443 2024-04-22T06:28:51.332 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.523+0000 7ff3ff00e700 1 Processor -- start 2024-04-22T06:28:51.332 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.523+0000 7ff3ff00e700 1 -- start start 2024-04-22T06:28:51.332 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3ff00e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3ff00e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f8108f60 con 0x7ff3f8108570 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3fcdaa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3fcdaa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40542/0 (socket says 172.21.15.105:40542) 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3fcdaa700 1 -- 172.21.15.105:0/2144023379 learned_addr learned my addr 172.21.15.105:0/2144023379 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3fcdaa700 1 -- 172.21.15.105:0/2144023379 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f8109770 con 0x7ff3f8108570 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.524+0000 7ff3fcdaa700 1 --2- 172.21.15.105:0/2144023379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0098a0 tx=0x7ff3ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a63be984b5d8eb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.525+0000 7ff3f77fe700 1 -- 172.21.15.105:0/2144023379 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3ec004160 con 0x7ff3f8108570 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.525+0000 7ff3f77fe700 1 -- 172.21.15.105:0/2144023379 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec004300 con 0x7ff3f8108570 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.525+0000 7ff3f77fe700 1 -- 172.21.15.105:0/2144023379 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3ec011a20 con 0x7ff3f8108570 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.525+0000 7ff3ff00e700 1 -- 172.21.15.105:0/2144023379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 msgr2=0x7ff3f8108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:51.333 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.525+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/2144023379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0098a0 tx=0x7ff3ec009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- 172.21.15.105:0/2144023379 shutdown_connections 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/2144023379 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f8108990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- 172.21.15.105:0/2144023379 >> 172.21.15.105:0/2144023379 conn(0x7ff3f8103b90 msgr2=0x7ff3f8105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- 172.21.15.105:0/2144023379 shutdown_connections 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- 172.21.15.105:0/2144023379 wait complete. 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 Processor -- start 2024-04-22T06:28:51.334 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- start start 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.526+0000 7ff3ff00e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f810cb20 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3fcdaa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3fcdaa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40544/0 (socket says 172.21.15.105:40544) 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3fcdaa700 1 -- 172.21.15.105:0/3230898449 learned_addr learned my addr 172.21.15.105:0/3230898449 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3fcdaa700 1 -- 172.21.15.105:0/3230898449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f807cf90 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3fcdaa700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009520 tx=0x7ff3ec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3ec0039e0 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f807b180 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3ec01b8f0 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.527+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3ec019770 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.528+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f807b6c0 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff3ec019a20 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f80640a0 con 0x7ff3f8108570 2024-04-22T06:28:51.335 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3f5ffb700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 0x7ff3e803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:51.336 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff3ec04e300 con 0x7ff3f8108570 2024-04-22T06:28:51.336 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3f7fff700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 0x7ff3e803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.529+0000 7ff3f7fff700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 0x7ff3e803bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4006fd0 tx=0x7ff3e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.538+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3ec01f030 con 0x7ff3f8108570 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.593+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ff3ec01a8f0 con 0x7ff3f8108570 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.747+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff3f807bfa0 con 0x7ff3f8108570 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.748+0000 7ff3f5ffb700 1 -- 172.21.15.105:0/3230898449 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff3ec0172b0 con 0x7ff3f8108570 2024-04-22T06:28:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 msgr2=0x7ff3e803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 0x7ff3e803bb10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4006fd0 tx=0x7ff3e4006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 msgr2=0x7ff3f807ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009520 tx=0x7ff3ec011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 shutdown_connections 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff3e8039650 0x7ff3e803bb10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 --2- 172.21.15.105:0/3230898449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3f8108570 0x7ff3f807ca50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 >> 172.21.15.105:0/3230898449 conn(0x7ff3f8103b90 msgr2=0x7ff3f81045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 shutdown_connections 2024-04-22T06:28:51.338 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:50.750+0000 7ff3ff00e700 1 -- 172.21.15.105:0/3230898449 wait complete. 2024-04-22T06:28:51.339 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-22T06:28:51.339 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-04-22T06:28:51.346 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-22T06:28:51.346 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-04-22T06:28:51.346 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-04-22T06:28:51.347 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph Dashboard is now available at: 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout: URL: https://smithi105:8443/ 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout: User: admin 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout: Password: 6npfb8k5hc 2024-04-22T06:28:51.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:51.349 INFO:teuthology.orchestra.run.smithi105.stdout:Saving cluster configuration to /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config directory 2024-04-22T06:28:51.349 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling autotune for osd_memory_target 2024-04-22T06:28:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:51 smithi105 ceph-mon[27916]: mgrmap e13: x(active, since 6s) 2024-04-22T06:28:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3230898449' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5eb55700 1 Processor -- start 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5eb55700 1 -- start start 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5eb55700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5eb55700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efc58107260 con 0x7efc58106870 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5c8f1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5c8f1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40560/0 (socket says 172.21.15.105:40560) 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5c8f1700 1 -- 172.21.15.105:0/3641160278 learned_addr learned my addr 172.21.15.105:0/3641160278 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.789+0000 7efc5c8f1700 1 -- 172.21.15.105:0/3641160278 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581098a0 con 0x7efc58106870 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5c8f1700 1 --2- 172.21.15.105:0/3641160278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0098a0 tx=0x7efc4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308c8334efc2c16b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:52.555 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc577fe700 1 -- 172.21.15.105:0/3641160278 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc4c004160 con 0x7efc58106870 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc577fe700 1 -- 172.21.15.105:0/3641160278 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4c004300 con 0x7efc58106870 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc577fe700 1 -- 172.21.15.105:0/3641160278 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc4c011a20 con 0x7efc58106870 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 -- 172.21.15.105:0/3641160278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 msgr2=0x7efc58106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 --2- 172.21.15.105:0/3641160278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0098a0 tx=0x7efc4c009bb0 comp rx=0 tx=0).stop 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 -- 172.21.15.105:0/3641160278 shutdown_connections 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 --2- 172.21.15.105:0/3641160278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58106c90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 -- 172.21.15.105:0/3641160278 >> 172.21.15.105:0/3641160278 conn(0x7efc581021c0 msgr2=0x7efc581045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 -- 172.21.15.105:0/3641160278 shutdown_connections 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.790+0000 7efc5eb55700 1 -- 172.21.15.105:0/3641160278 wait complete. 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5eb55700 1 Processor -- start 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5eb55700 1 -- start start 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5eb55700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5eb55700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efc5810cad0 con 0x7efc58106870 2024-04-22T06:28:52.556 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5c8f1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5c8f1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40568/0 (socket says 172.21.15.105:40568) 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.791+0000 7efc5c8f1700 1 -- 172.21.15.105:0/952349404 learned_addr learned my addr 172.21.15.105:0/952349404 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc5c8f1700 1 -- 172.21.15.105:0/952349404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc58109660 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc5c8f1700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009520 tx=0x7efc4c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc4c0039e0 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4c01b8f0 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc58107850 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc4c019770 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.792+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc58107d90 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.793+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7efc4c019a20 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.793+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc580640a0 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.793+0000 7efc55ffb700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 0x7efc4803bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.793+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efc4c04e2e0 con 0x7efc58106870 2024-04-22T06:28:52.557 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.793+0000 7efc57fff700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 0x7efc4803bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.794+0000 7efc57fff700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 0x7efc4803bbb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efc44006fd0 tx=0x7efc44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.798+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc4c01f030 con 0x7efc58106870 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.995+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7efc581086e0 con 0x7efc58106870 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.995+0000 7efc55ffb700 1 -- 172.21.15.105:0/952349404 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7efc581086e0 con 0x7efc58106870 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.997+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 msgr2=0x7efc4803bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.997+0000 7efc5eb55700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 0x7efc4803bbb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efc44006fd0 tx=0x7efc44006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.997+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 msgr2=0x7efc58109120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.997+0000 7efc5eb55700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009520 tx=0x7efc4c011aa0 comp rx=0 tx=0).stop 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 shutdown_connections 2024-04-22T06:28:52.558 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efc480396f0 0x7efc4803bbb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:52.559 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 --2- 172.21.15.105:0/952349404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc58106870 0x7efc58109120 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:52.559 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 >> 172.21.15.105:0/952349404 conn(0x7efc581021c0 msgr2=0x7efc58102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:52.559 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 shutdown_connections 2024-04-22T06:28:52.559 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:51.998+0000 7efc5eb55700 1 -- 172.21.15.105:0/952349404 wait complete. 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.029+0000 7f4bf9e27700 1 Processor -- start 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.029+0000 7f4bf9e27700 1 -- start start 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.029+0000 7f4bf9e27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.029+0000 7f4bf9e27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf4108380 con 0x7f4bf4107990 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40576/0 (socket says 172.21.15.105:40576) 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf37fe700 1 -- 172.21.15.105:0/1314011693 learned_addr learned my addr 172.21.15.105:0/1314011693 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf37fe700 1 -- 172.21.15.105:0/1314011693 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf41086b0 con 0x7f4bf4107990 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf37fe700 1 --2- 172.21.15.105:0/1314011693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009e80 tx=0x7f4be400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754d46b77388181c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf27fc700 1 -- 172.21.15.105:0/1314011693 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be4017070 con 0x7f4bf4107990 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf27fc700 1 -- 172.21.15.105:0/1314011693 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be400b8c0 con 0x7f4bf4107990 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.030+0000 7f4bf27fc700 1 -- 172.21.15.105:0/1314011693 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be4004490 con 0x7f4bf4107990 2024-04-22T06:28:53.945 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 -- 172.21.15.105:0/1314011693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 msgr2=0x7f4bf4107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/1314011693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009e80 tx=0x7f4be400b330 comp rx=0 tx=0).stop 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 -- 172.21.15.105:0/1314011693 shutdown_connections 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/1314011693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf4107db0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 -- 172.21.15.105:0/1314011693 >> 172.21.15.105:0/1314011693 conn(0x7f4bf4102210 msgr2=0x7f4bf4104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 -- 172.21.15.105:0/1314011693 shutdown_connections 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 -- 172.21.15.105:0/1314011693 wait complete. 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.031+0000 7f4bf9e27700 1 Processor -- start 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf9e27700 1 -- start start 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf9e27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf9e27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf410ba60 con 0x7f4bf4107990 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf37fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40592/0 (socket says 172.21.15.105:40592) 2024-04-22T06:28:53.946 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf37fe700 1 -- 172.21.15.105:0/3028583291 learned_addr learned my addr 172.21.15.105:0/3028583291 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf37fe700 1 -- 172.21.15.105:0/3028583291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf4192b10 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf37fe700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4be4004220 tx=0x7f4be400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be4020020 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be4003b50 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be4017040 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf4190d50 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.032+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf4191290 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.033+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f4be4023020 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.033+0000 7f4bf0ff9700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 0x7f4bdc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.033+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4be400f050 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.033+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf40640a0 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.034+0000 7f4bf2ffd700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 0x7f4bdc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.034+0000 7f4bf2ffd700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 0x7f4bdc03bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4be8006fd0 tx=0x7f4be8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.039+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4be402a020 con 0x7f4bf4107990 2024-04-22T06:28:53.947 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.348+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f4bf41961c0 con 0x7f4bf4107990 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.351+0000 7f4bf0ff9700 1 -- 172.21.15.105:0/3028583291 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f4be4007570 con 0x7f4bf4107990 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 msgr2=0x7f4bdc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 0x7f4bdc03bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4be8006fd0 tx=0x7f4be8006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 msgr2=0x7f4bf41925d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4be4004220 tx=0x7f4be400bd30 comp rx=0 tx=0).stop 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 shutdown_connections 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4bdc0396b0 0x7f4bdc03bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 --2- 172.21.15.105:0/3028583291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4bf4107990 0x7f4bf41925d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 >> 172.21.15.105:0/3028583291 conn(0x7f4bf4102210 msgr2=0x7f4bf4103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 shutdown_connections 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-04-22T06:28:53.353+0000 7f4bf9e27700 1 -- 172.21.15.105:0/3028583291 wait complete. 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:28:53.948 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout:Or, if you are only running a single cluster on this host: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: ceph telemetry on 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout:For more information see: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:28:53.949 INFO:teuthology.orchestra.run.smithi105.stdout:Bootstrap complete. 2024-04-22T06:28:53.984 INFO:tasks.cephadm:Fetching config... 2024-04-22T06:28:53.985 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:28:53.985 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T06:28:54.002 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T06:28:54.002 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:28:54.002 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T06:28:54.058 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T06:28:54.058 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:28:54.058 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T06:28:54.122 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T06:28:54.122 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:28:54.122 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T06:28:54.177 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T06:28:54.177 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:28:54.252 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:54.263 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:28:54.300 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:54.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T06:28:54.350 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRXH/dku/2nmsj6Ri8wz6vxXIJpyR7HjNKMi3U/iUeErG/MLMY/NA3KsAWWOfU8MLHu8UA+xgA3gihTzx3trO5cW4p4Z3e1yuRaL8Kjj4Qr5GLE1z7R5CxnvT3JB3g5r7W9kFc+wyBIDO7NQf3iGr/9ZpTrbTZiTMP3/Qm4Y4jBEr9vCibH8kdfhURzXoyf5lyGVjfbmu2VyeTtDn5LLXqhVNfgLG0Was2wjVIDBOsqikzF9Co9xwe86EHeMMtUGO9L51FdlBXKuZW6n2OhvBOqDm0VDuxHFIVIXdeKL7Ht+PpES6c7PyYgdJqCHhBQVhuKOFsUQ84smxUdCUe/nYnKD8fRw8AN6qUXeE/7VJlKLQXUAZz4/v3dW/qa/7mD62XeWEfU9izuogIHtUpcySWD5jFV5N0X/RMDOLidRJ+XleeByu6qbpVU+So6inU3RWHdA9hyjAb0E5O8SO59F4HL3gRvL+1sUtHVY9xhhd3lQ1QNzxfq4Te/qw1ZJghUj0= ceph-48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:28:54.362 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T06:28:54.595 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:28:54.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:54 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3028583291' entity='client.admin' 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.137+0000 7f5656faf700 1 -- 172.21.15.105:0/226399516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 msgr2=0x7f56501053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.137+0000 7f5656faf700 1 --2- 172.21.15.105:0/226399516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f56501053d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5644009910 tx=0x7f5644009c20 comp rx=0 tx=0).stop 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- 172.21.15.105:0/226399516 shutdown_connections 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 --2- 172.21.15.105:0/226399516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f56501053d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- 172.21.15.105:0/226399516 >> 172.21.15.105:0/226399516 conn(0x7f56500fc950 msgr2=0x7f56500fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:57.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- 172.21.15.105:0/226399516 shutdown_connections 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- 172.21.15.105:0/226399516 wait complete. 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 Processor -- start 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- start start 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5656faf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56501083f0 con 0x7f5650102fe0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5654d4b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5654d4b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54744/0 (socket says 172.21.15.105:54744) 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.138+0000 7f5654d4b700 1 -- 172.21.15.105:0/2274395036 learned_addr learned my addr 172.21.15.105:0/2274395036 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f5654d4b700 1 -- 172.21.15.105:0/2274395036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5650102da0 con 0x7f5650102fe0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f5654d4b700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5644005b40 tx=0x7f5644004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5644027070 con 0x7f5650102fe0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5650100f90 con 0x7f5650102fe0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f564400be10 con 0x7f5650102fe0 2024-04-22T06:28:57.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5644022d80 con 0x7f5650102fe0 2024-04-22T06:28:57.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.139+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56501014d0 con 0x7f5650102fe0 2024-04-22T06:28:57.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.140+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f564402a070 con 0x7f5650102fe0 2024-04-22T06:28:57.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.140+0000 7f564dffb700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 0x7f564003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:28:57.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.140+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f564401e030 con 0x7f5650102fe0 2024-04-22T06:28:57.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.140+0000 7f564ffff700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 0x7f564003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:28:57.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.140+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5650108b70 con 0x7f5650102fe0 2024-04-22T06:28:57.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.141+0000 7f564ffff700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 0x7f564003bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f563c006fd0 tx=0x7f563c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:28:57.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.145+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5644013070 con 0x7f5650102fe0 2024-04-22T06:28:57.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.384+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f5650190d60 con 0x7f5650102fe0 2024-04-22T06:28:57.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.386+0000 7f564dffb700 1 -- 172.21.15.105:0/2274395036 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f5644035030 con 0x7f5650102fe0 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 msgr2=0x7f564003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 0x7f564003bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f563c006fd0 tx=0x7f563c006e40 comp rx=0 tx=0).stop 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 msgr2=0x7f5650102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5644005b40 tx=0x7f5644004f80 comp rx=0 tx=0).stop 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 shutdown_connections 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5640039650 0x7f564003bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 --2- 172.21.15.105:0/2274395036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5650102fe0 0x7f5650102860 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:28:57.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 >> 172.21.15.105:0/2274395036 conn(0x7f56500fc950 msgr2=0x7f56500fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:28:57.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 shutdown_connections 2024-04-22T06:28:57.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:28:57.389+0000 7f5656faf700 1 -- 172.21.15.105:0/2274395036 wait complete. 2024-04-22T06:28:58.021 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T06:28:58.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T06:28:58.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:58 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2274395036' entity='client.admin' 2024-04-22T06:28:58.688 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:28:59.900 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:59.901 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:59.901 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:28:59.901 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:28:59.901 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:28:59.901 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:28:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:00.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 -- 172.21.15.105:0/552179225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 msgr2=0x7fd7b4105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:00.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 --2- 172.21.15.105:0/552179225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4105340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009910 tx=0x7fd7b0009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 -- 172.21.15.105:0/552179225 shutdown_connections 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 --2- 172.21.15.105:0/552179225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4105340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 -- 172.21.15.105:0/552179225 >> 172.21.15.105:0/552179225 conn(0x7fd7b40fc940 msgr2=0x7fd7b40feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 -- 172.21.15.105:0/552179225 shutdown_connections 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.220+0000 7fd7bc280700 1 -- 172.21.15.105:0/552179225 wait complete. 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7bc280700 1 Processor -- start 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7bc280700 1 -- start start 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7bc280700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7bc280700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b41083a0 con 0x7fd7b4102f50 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7ba01c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7ba01c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54764/0 (socket says 172.21.15.105:54764) 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.221+0000 7fd7ba01c700 1 -- 172.21.15.105:0/939528688 learned_addr learned my addr 172.21.15.105:0/939528688 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:00.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7ba01c700 1 -- 172.21.15.105:0/939528688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b4198010 con 0x7fd7b4102f50 2024-04-22T06:29:00.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7ba01c700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005b40 tx=0x7fd7b0004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b0027070 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b4198300 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7b000be10 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b0022d80 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.222+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b419af90 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.223+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fd7b002a070 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.223+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b4108b20 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.223+0000 7fd7ab7fe700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 0x7fd7a003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.223+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd7b001e030 con 0x7fd7b4102f50 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.223+0000 7fd7b981b700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 0x7fd7a003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:00.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.224+0000 7fd7b981b700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 0x7fd7a003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4006fd0 tx=0x7fd7a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:00.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.228+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7b0013070 con 0x7fd7b4102f50 2024-04-22T06:29:00.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.430+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fd7b402e190 con 0x7fd7a0039650 2024-04-22T06:29:00.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.433+0000 7fd7ab7fe700 1 -- 172.21.15.105:0/939528688 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd7b402e190 con 0x7fd7a0039650 2024-04-22T06:29:00.434 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 msgr2=0x7fd7a003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 0x7fd7a003bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4006fd0 tx=0x7fd7a4006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 msgr2=0x7fd7b4197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005b40 tx=0x7fd7b0004f80 comp rx=0 tx=0).stop 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 shutdown_connections 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd7a0039650 0x7fd7a003bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 --2- 172.21.15.105:0/939528688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7b4102f50 0x7fd7b4197ad0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.435+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 >> 172.21.15.105:0/939528688 conn(0x7fd7b40fc940 msgr2=0x7fd7b40fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.436+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 shutdown_connections 2024-04-22T06:29:00.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:00.436+0000 7fd7bc280700 1 -- 172.21.15.105:0/939528688 wait complete. 2024-04-22T06:29:01.290 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2024-04-22T06:29:01.291 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:29:01.291 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:29:01.310 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:29:01.310 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:01.367 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2024-04-22T06:29:01.368 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch host add smithi112 2024-04-22T06:29:01.597 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:01.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:01.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:02.601 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:02 smithi105 ceph-mon[27916]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:02.602 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:02 smithi105 ceph-mon[27916]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 -- 172.21.15.105:0/82487195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 msgr2=0x7f7de41088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 --2- 172.21.15.105:0/82487195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de41088c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009910 tx=0x7f7dd8009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 -- 172.21.15.105:0/82487195 shutdown_connections 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 --2- 172.21.15.105:0/82487195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de41088c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 -- 172.21.15.105:0/82487195 >> 172.21.15.105:0/82487195 conn(0x7f7de40fdf60 msgr2=0x7f7de4100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 -- 172.21.15.105:0/82487195 shutdown_connections 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.059+0000 7f7dea307700 1 -- 172.21.15.105:0/82487195 wait complete. 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7dea307700 1 Processor -- start 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7dea307700 1 -- start start 2024-04-22T06:29:03.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7dea307700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7dea307700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410c940 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7de3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7de3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54782/0 (socket says 172.21.15.105:54782) 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.060+0000 7f7de3fff700 1 -- 172.21.15.105:0/1572109105 learned_addr learned my addr 172.21.15.105:0/1572109105 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7de3fff700 1 -- 172.21.15.105:0/1572109105 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de41983d0 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7de3fff700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80056d0 tx=0x7f7dd80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dd8026070 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de41986c0 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd800bba0 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7dd8003920 con 0x7f7de41084e0 2024-04-22T06:29:03.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.061+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de419c760 con 0x7f7de41084e0 2024-04-22T06:29:03.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f7dd8029070 con 0x7f7de41084e0 2024-04-22T06:29:03.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de4103b90 con 0x7f7de41084e0 2024-04-22T06:29:03.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7de17fa700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 0x7f7dcc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:03.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7dd801e070 con 0x7f7de41084e0 2024-04-22T06:29:03.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7de37fe700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 0x7f7dcc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:03.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.062+0000 7f7de37fe700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 0x7f7dcc03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4006fd0 tx=0x7f7dd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:03.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.067+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7dd8013070 con 0x7f7de41084e0 2024-04-22T06:29:03.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:03.272+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}) -- 0x7f7de41991f0 con 0x7f7dcc039650 2024-04-22T06:29:04.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:03 smithi105 ceph-mon[27916]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:05.122 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:04 smithi105 ceph-mon[27916]: Deploying cephadm binary to smithi112 2024-04-22T06:29:05.122 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:04 smithi105 ceph-mon[27916]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:07.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.337+0000 7f7de17fa700 1 -- 172.21.15.105:0/1572109105 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7de41991f0 con 0x7f7dcc039650 2024-04-22T06:29:07.337 INFO:teuthology.orchestra.run.smithi105.stdout:Added host 'smithi112' with addr '172.21.15.112' 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 msgr2=0x7f7dcc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 0x7f7dcc03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4006fd0 tx=0x7f7dd4006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 msgr2=0x7f7de4197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7dd80056d0 tx=0x7f7dd80057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 shutdown_connections 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7dcc039650 0x7f7dcc03bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 --2- 172.21.15.105:0/1572109105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7de41084e0 0x7f7de4197e90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 >> 172.21.15.105:0/1572109105 conn(0x7f7de40fdf60 msgr2=0x7f7de40fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 shutdown_connections 2024-04-22T06:29:07.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:07.340+0000 7f7dea307700 1 -- 172.21.15.105:0/1572109105 wait complete. 2024-04-22T06:29:07.899 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:07 smithi105 ceph-mon[27916]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:07.899 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:07.900 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:07.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:29:08.161 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:08.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:08 smithi105 ceph-mon[27916]: Added host smithi112 2024-04-22T06:29:08.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:08 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:08.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:08 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:09.641 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:09 smithi105 ceph-mon[27916]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:09.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 -- 172.21.15.105:0/1000352545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 msgr2=0x7fe2981088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:09.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 --2- 172.21.15.105:0/1000352545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe2981088a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe288009910 tx=0x7fe288009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:09.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 -- 172.21.15.105:0/1000352545 shutdown_connections 2024-04-22T06:29:09.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 --2- 172.21.15.105:0/1000352545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe2981088a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:09.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 -- 172.21.15.105:0/1000352545 >> 172.21.15.105:0/1000352545 conn(0x7fe2980fdf60 msgr2=0x7fe298100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:09.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 -- 172.21.15.105:0/1000352545 shutdown_connections 2024-04-22T06:29:09.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.661+0000 7fe29de69700 1 -- 172.21.15.105:0/1000352545 wait complete. 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe29de69700 1 Processor -- start 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe29de69700 1 -- start start 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe29de69700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe29de69700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe29810c920 con 0x7fe2981084c0 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe2977fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe2977fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53030/0 (socket says 172.21.15.105:53030) 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.662+0000 7fe2977fe700 1 -- 172.21.15.105:0/1207177507 learned_addr learned my addr 172.21.15.105:0/1207177507 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:09.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe2977fe700 1 -- 172.21.15.105:0/1207177507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe298076c30 con 0x7fe2981084c0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe2977fe700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe288006010 tx=0x7fe288005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe288026070 con 0x7fe2981084c0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2980752e0 con 0x7fe2981084c0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28800bc00 con 0x7fe2981084c0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe288003910 con 0x7fe2981084c0 2024-04-22T06:29:09.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.663+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2980757a0 con 0x7fe2981084c0 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.664+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fe288029070 con 0x7fe2981084c0 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.664+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe298103b70 con 0x7fe2981084c0 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.664+0000 7fe294ff9700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 0x7fe28003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.664+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe28801e070 con 0x7fe2981084c0 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.664+0000 7fe296ffd700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 0x7fe28003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:09.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.665+0000 7fe296ffd700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 0x7fe28003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:09.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.669+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe288013070 con 0x7fe2981084c0 2024-04-22T06:29:09.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.873+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2980762e0 con 0x7fe280039650 2024-04-22T06:29:09.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.874+0000 7fe294ff9700 1 -- 172.21.15.105:0/1207177507 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fe2980762e0 con 0x7fe280039650 2024-04-22T06:29:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:29:09.873 INFO:teuthology.orchestra.run.smithi105.stdout:[{"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}] 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 msgr2=0x7fe28003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 0x7fe28003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe28c006fd0 tx=0x7fe28c006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 msgr2=0x7fe298074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe288006010 tx=0x7fe288005780 comp rx=0 tx=0).stop 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 shutdown_connections 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe280039650 0x7fe28003bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 --2- 172.21.15.105:0/1207177507 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2981084c0 0x7fe298074d40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 >> 172.21.15.105:0/1207177507 conn(0x7fe2980fdf60 msgr2=0x7fe29810a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 shutdown_connections 2024-04-22T06:29:09.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:09.876+0000 7fe29de69700 1 -- 172.21.15.105:0/1207177507 wait complete. 2024-04-22T06:29:10.455 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-04-22T06:29:10.455 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:29:10.455 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-04-22T06:29:10.474 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:29:10.475 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:10.532 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-04-22T06:29:10.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch host add smithi177 2024-04-22T06:29:10.760 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:11.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:10 smithi105 ceph-mon[27916]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:29:11.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:11.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:10 smithi105 ceph-mon[27916]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:12.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- 172.21.15.105:0/3939529974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 msgr2=0x7f8c7c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:12.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 --2- 172.21.15.105:0/3939529974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c1039c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009910 tx=0x7f8c70009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:12.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- 172.21.15.105:0/3939529974 shutdown_connections 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 --2- 172.21.15.105:0/3939529974 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c1039c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- 172.21.15.105:0/3939529974 >> 172.21.15.105:0/3939529974 conn(0x7f8c7c078010 msgr2=0x7f8c7c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- 172.21.15.105:0/3939529974 shutdown_connections 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- 172.21.15.105:0/3939529974 wait complete. 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 Processor -- start 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.331+0000 7f8c845d0700 1 -- start start 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c845d0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c845d0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7c10c980 con 0x7f8c7c1035e0 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c8236c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c8236c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53050/0 (socket says 172.21.15.105:53050) 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c8236c700 1 -- 172.21.15.105:0/2059235556 learned_addr learned my addr 172.21.15.105:0/2059235556 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c8236c700 1 -- 172.21.15.105:0/2059235556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7c1984f0 con 0x7f8c7c1035e0 2024-04-22T06:29:12.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c8236c700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c700056d0 tx=0x7f8c700057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:12.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c70026070 con 0x7f8c7c1035e0 2024-04-22T06:29:12.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7c198780 con 0x7f8c7c1035e0 2024-04-22T06:29:12.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c70004370 con 0x7f8c7c1035e0 2024-04-22T06:29:12.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.332+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c700039b0 con 0x7f8c7c1035e0 2024-04-22T06:29:12.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.333+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7c198be0 con 0x7f8c7c1035e0 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.333+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f8c70029070 con 0x7f8c7c1035e0 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.333+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7c06a750 con 0x7f8c7c1035e0 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.333+0000 7f8c6f7fe700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 0x7f8c6803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.334+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8c7001e070 con 0x7f8c7c1035e0 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.334+0000 7f8c81b6b700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 0x7f8c6803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:12.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.334+0000 7f8c81b6b700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 0x7f8c6803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8c78006fd0 tx=0x7f8c78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:12.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.339+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c70013070 con 0x7f8c7c1035e0 2024-04-22T06:29:12.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:12.540+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) -- 0x7f8c7c199330 con 0x7f8c68039650 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:29:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:14 smithi105 ceph-mon[27916]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:29:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:14 smithi105 ceph-mon[27916]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:14 smithi105 ceph-mon[27916]: Deploying cephadm binary to smithi177 2024-04-22T06:29:14.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:14 smithi105 ceph-mon[27916]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:14.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:14 smithi105 ceph-mon[27916]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:29:15.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:15 smithi105 ceph-mon[27916]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:16.681 INFO:teuthology.orchestra.run.smithi105.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-04-22T06:29:16.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.682+0000 7f8c6f7fe700 1 -- 172.21.15.105:0/2059235556 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8c7c199330 con 0x7f8c68039650 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 msgr2=0x7f8c6803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 0x7f8c6803bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8c78006fd0 tx=0x7f8c78006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 msgr2=0x7f8c7c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8c700056d0 tx=0x7f8c700057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 shutdown_connections 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8c68039650 0x7f8c6803bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 --2- 172.21.15.105:0/2059235556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c7c1035e0 0x7f8c7c197fb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 >> 172.21.15.105:0/2059235556 conn(0x7f8c7c078010 msgr2=0x7f8c7c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 shutdown_connections 2024-04-22T06:29:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:16.684+0000 7f8c845d0700 1 -- 172.21.15.105:0/2059235556 wait complete. 2024-04-22T06:29:17.224 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: Added host smithi177 2024-04-22T06:29:17.225 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:17.226 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T06:29:17.455 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:18.613 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:18 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:18.613 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:18 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:19.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- 172.21.15.105:0/2177164486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 msgr2=0x7f2fb8108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:19.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/2177164486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8108990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009910 tx=0x7f2fa8009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:19.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- 172.21.15.105:0/2177164486 shutdown_connections 2024-04-22T06:29:19.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/2177164486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8108990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:19.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- 172.21.15.105:0/2177164486 >> 172.21.15.105:0/2177164486 conn(0x7f2fb80fc5c0 msgr2=0x7f2fb80fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:19.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- 172.21.15.105:0/2177164486 shutdown_connections 2024-04-22T06:29:19.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- 172.21.15.105:0/2177164486 wait complete. 2024-04-22T06:29:19.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 Processor -- start 2024-04-22T06:29:19.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.126+0000 7f2fbdb95700 1 -- start start 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fbdb95700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fbdb95700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb810c990 con 0x7f2fb81085b0 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb77fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb77fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45876/0 (socket says 172.21.15.105:45876) 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb77fe700 1 -- 172.21.15.105:0/3444467918 learned_addr learned my addr 172.21.15.105:0/3444467918 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:19.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb77fe700 1 -- 172.21.15.105:0/3444467918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8103970 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb77fe700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80056d0 tx=0x7f2fa80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fa8027070 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa800be10 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb8101fc0 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.127+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2fa8022d20 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.128+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb8102480 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.128+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f2fa802a070 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.128+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80640a0 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.128+0000 7f2fb57fa700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 0x7f2fa403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.128+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2fa801e030 con 0x7f2fb81085b0 2024-04-22T06:29:19.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.129+0000 7f2faeffd700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 0x7f2fa403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:19.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.129+0000 7f2faeffd700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 0x7f2fa403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0006fd0 tx=0x7f2fa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:19.133 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.134+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fa8013070 con 0x7f2fb81085b0 2024-04-22T06:29:19.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.346+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fb81026b0 con 0x7f2fa40396b0 2024-04-22T06:29:19.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.347+0000 7f2fb57fa700 1 -- 172.21.15.105:0/3444467918 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f2fb81026b0 con 0x7f2fa40396b0 2024-04-22T06:29:19.347 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:29:19.347 INFO:teuthology.orchestra.run.smithi105.stdout:[{"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-04-22T06:29:19.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.349+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 msgr2=0x7f2fa403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.349+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 0x7f2fa403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0006fd0 tx=0x7f2fa0006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.349+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 msgr2=0x7f2fb8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.349+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80056d0 tx=0x7f2fa80057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 shutdown_connections 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fa40396b0 0x7f2fa403bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 --2- 172.21.15.105:0/3444467918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fb81085b0 0x7f2fb8101a80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 >> 172.21.15.105:0/3444467918 conn(0x7f2fb80fc5c0 msgr2=0x7f2fb80fc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 shutdown_connections 2024-04-22T06:29:19.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:19.350+0000 7f2fbdb95700 1 -- 172.21.15.105:0/3444467918 wait complete. 2024-04-22T06:29:19.451 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:19 smithi105 ceph-mon[27916]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:20.020 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T06:29:20.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T06:29:20.258 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:20.572 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:20.572 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:20 smithi105 ceph-mon[27916]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:29:21.461 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:21 smithi105 ceph-mon[27916]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:21.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 -- 172.21.15.105:0/3157327110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 msgr2=0x7f1fd01088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3157327110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd01088a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009910 tx=0x7f1fcc009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 -- 172.21.15.105:0/3157327110 shutdown_connections 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3157327110 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd01088a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 -- 172.21.15.105:0/3157327110 >> 172.21.15.105:0/3157327110 conn(0x7f1fd00fdf60 msgr2=0x7f1fd0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 -- 172.21.15.105:0/3157327110 shutdown_connections 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.669+0000 7f1fd7723700 1 -- 172.21.15.105:0/3157327110 wait complete. 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd7723700 1 Processor -- start 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd7723700 1 -- start start 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd7723700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd7723700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd010c920 con 0x7f1fd01084c0 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd54bf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:21.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd54bf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45894/0 (socket says 172.21.15.105:45894) 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd54bf700 1 -- 172.21.15.105:0/3683906140 learned_addr learned my addr 172.21.15.105:0/3683906140 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd54bf700 1 -- 172.21.15.105:0/3683906140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd01983b0 con 0x7f1fd01084c0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd54bf700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0056d0 tx=0x7f1fcc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fcc026070 con 0x7f1fd01084c0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fcc00bba0 con 0x7f1fd01084c0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd01986a0 con 0x7f1fd01084c0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.670+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fcc003920 con 0x7f1fd01084c0 2024-04-22T06:29:21.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.671+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd019c740 con 0x7f1fd01084c0 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.671+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f1fcc029070 con 0x7f1fd01084c0 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.671+0000 7f1fc67fc700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 0x7f1fbc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.672+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1fcc01e070 con 0x7f1fd01084c0 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.672+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd0103b70 con 0x7f1fd01084c0 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.672+0000 7f1fd4cbe700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 0x7f1fbc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:21.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.672+0000 7f1fd4cbe700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 0x7f1fbc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0006fd0 tx=0x7f1fc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:21.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.677+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fcc030070 con 0x7f1fd01084c0 2024-04-22T06:29:21.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:21.880+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1fd00640d0 con 0x7f1fd01084c0 2024-04-22T06:29:22.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.293+0000 7f1fc67fc700 1 -- 172.21.15.105:0/3683906140 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f1fd00640d0 con 0x7f1fd01084c0 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 msgr2=0x7f1fbc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 0x7f1fbc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0006fd0 tx=0x7f1fc0006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 msgr2=0x7f1fd0197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0056d0 tx=0x7f1fcc0057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 shutdown_connections 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1fbc039650 0x7f1fbc03bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:22.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 --2- 172.21.15.105:0/3683906140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1fd01084c0 0x7f1fd0197e70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:22.297 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 >> 172.21.15.105:0/3683906140 conn(0x7f1fd00fdf60 msgr2=0x7f1fd00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:22.297 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 shutdown_connections 2024-04-22T06:29:22.297 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:22.296+0000 7f1fd7723700 1 -- 172.21.15.105:0/3683906140 wait complete. 2024-04-22T06:29:22.297 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2024-04-22T06:29:22.666 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:22 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3683906140' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:29:23.001 INFO:tasks.cephadm:Adding mon.a on smithi105 2024-04-22T06:29:23.001 INFO:tasks.cephadm:Adding mon.b on smithi112 2024-04-22T06:29:23.001 INFO:tasks.cephadm:Adding mon.c on smithi177 2024-04-22T06:29:23.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c' 2024-04-22T06:29:23.231 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:23.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3683906140' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:23.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- 172.21.15.177:0/2309790587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 msgr2=0x7f3c1c101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 --2- 172.21.15.177:0/2309790587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c101960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009910 tx=0x7f3c18009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- 172.21.15.177:0/2309790587 shutdown_connections 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 --2- 172.21.15.177:0/2309790587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c101960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- 172.21.15.177:0/2309790587 >> 172.21.15.177:0/2309790587 conn(0x7f3c1c078010 msgr2=0x7f3c1c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- 172.21.15.177:0/2309790587 shutdown_connections 2024-04-22T06:29:24.625 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- 172.21.15.177:0/2309790587 wait complete. 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 Processor -- start 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- start start 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.625+0000 7f3c24237700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1c10b800 con 0x7f3c1c101580 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c21fd3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c21fd3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.177:50474/0 (socket says 172.21.15.177:50474) 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c21fd3700 1 -- 172.21.15.177:0/2966816054 learned_addr learned my addr 172.21.15.177:0/2966816054 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:29:24.626 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c21fd3700 1 -- 172.21.15.177:0/2966816054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1c199280 con 0x7f3c1c101580 2024-04-22T06:29:24.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c21fd3700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c180056d0 tx=0x7f3c180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:24.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c18026070 con 0x7f3c1c101580 2024-04-22T06:29:24.627 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1c199570 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1800bba0 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1c199a30 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.626+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c18003910 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.627+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f3c18029070 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.627+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c1c102a60 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.628+0000 7f3c12ffd700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 0x7f3c0803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.628+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3c1801e070 con 0x7f3c1c101580 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.628+0000 7f3c217d2700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 0x7f3c0803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:24.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.628+0000 7f3c217d2700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 0x7f3c0803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c006fd0 tx=0x7f3c0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:24.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.633+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c18013070 con 0x7f3c1c101580 2024-04-22T06:29:24.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:24 smithi105 ceph-mon[27916]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:24.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:24 smithi105 ceph-mon[27916]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:29:24.844 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.845+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}) -- 0x7f3c1c0640a0 con 0x7f3c08039650 2024-04-22T06:29:24.851 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.852+0000 7f3c12ffd700 1 -- 172.21.15.177:0/2966816054 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3c1c0640a0 con 0x7f3c08039650 2024-04-22T06:29:24.851 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 msgr2=0x7f3c0803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 0x7f3c0803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c006fd0 tx=0x7f3c0c006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 msgr2=0x7f3c1c198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3c180056d0 tx=0x7f3c180057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 shutdown_connections 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3c08039650 0x7f3c0803bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:24.853 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 --2- 172.21.15.177:0/2966816054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c1c101580 0x7f3c1c198d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:24.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 >> 172.21.15.177:0/2966816054 conn(0x7f3c1c078010 msgr2=0x7f3c1c102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:24.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 shutdown_connections 2024-04-22T06:29:24.854 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:24.854+0000 7f3c24237700 1 -- 172.21.15.177:0/2966816054 wait complete. 2024-04-22T06:29:25.355 DEBUG:teuthology.orchestra.run.smithi112:mon.b> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T06:29:25.358 DEBUG:teuthology.orchestra.run.smithi177:mon.c> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T06:29:25.360 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:29:25.360 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:29:25.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:25.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:26.486 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:26 smithi105 ceph-mon[27916]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:26 smithi105 ceph-mon[27916]: Saving service mon spec with placement smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c;count:3 2024-04-22T06:29:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:26 smithi105 ceph-mon[27916]: Deploying daemon mon.c on smithi177 2024-04-22T06:29:27.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:27 smithi105 ceph-mon[27916]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:29.001 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/3567174613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 msgr2=0x7fa7e4101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:29.001 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/3567174613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4101990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009910 tx=0x7fa7e0009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:29.001 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/3567174613 shutdown_connections 2024-04-22T06:29:29.001 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/3567174613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4101990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/3567174613 >> 172.21.15.177:0/3567174613 conn(0x7fa7e40fce80 msgr2=0x7fa7e40ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/3567174613 shutdown_connections 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:28.999+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/3567174613 wait complete. 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7ec1b3700 1 Processor -- start 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7ec1b3700 1 -- start start 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7ec1b3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7ec1b3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e410b830 con 0x7fa7e41015b0 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7e9f4f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7e9f4f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.177:50500/0 (socket says 172.21.15.177:50500) 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.000+0000 7fa7e9f4f700 1 -- 172.21.15.177:0/1057787542 learned_addr learned my addr 172.21.15.177:0/1057787542 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7e9f4f700 1 -- 172.21.15.177:0/1057787542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e41972e0 con 0x7fa7e41015b0 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7e9f4f700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00056d0 tx=0x7fa7e00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:29.002 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e0026070 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e000bba0 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e41975d0 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e419b670 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.001+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7e0003920 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.002+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4102a90 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.002+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fa7e0029070 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.002+0000 7fa7daffd700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 0x7fa7d003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.002+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa7e001e070 con 0x7fa7e41015b0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.002+0000 7fa7e974e700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 0x7fa7d003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:29.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.003+0000 7fa7e974e700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 0x7fa7d003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4006fd0 tx=0x7fa7d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:29.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.008+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7e0013070 con 0x7fa7e41015b0 2024-04-22T06:29:29.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.314+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa7e4198100 con 0x7fa7e41015b0 2024-04-22T06:29:29.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.314+0000 7fa7daffd700 1 -- 172.21.15.177:0/1057787542 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fa7e00282c0 con 0x7fa7e41015b0 2024-04-22T06:29:29.314 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:29:29.314 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","modified":"2024-04-22T06:27:38.293503Z","created":"2024-04-22T06:27:38.293503Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 msgr2=0x7fa7d003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 0x7fa7d003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4006fd0 tx=0x7fa7d4006e40 comp rx=0 tx=0).stop 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 msgr2=0x7fa7e4196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00056d0 tx=0x7fa7e00057b0 comp rx=0 tx=0).stop 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 shutdown_connections 2024-04-22T06:29:29.316 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7d0039650 0x7fa7d003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:29.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 --2- 172.21.15.177:0/1057787542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7e41015b0 0x7fa7e4196da0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:29.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 >> 172.21.15.177:0/1057787542 conn(0x7fa7e40fce80 msgr2=0x7fa7e4102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:29.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 shutdown_connections 2024-04-22T06:29:29.317 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:29.317+0000 7fa7ec1b3700 1 -- 172.21.15.177:0/1057787542 wait complete. 2024-04-22T06:29:29.317 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-04-22T06:29:29.886 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:29 smithi105 ceph-mon[27916]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:29.886 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:29 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/1057787542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:30.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 podman[31327]: 2024-04-22 06:29:30.0817431 +0000 UTC m=+0.018021574 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 podman[31327]: 2024-04-22 06:29:30.378426547 +0000 UTC m=+0.314705019 container init f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 podman[31327]: 2024-04-22 06:29:30.381532257 +0000 UTC m=+0.317810717 container start f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev), process ceph-mon, pid 2 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pidfile_write: ignore empty --pid-file 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: load: jerasure load: lrc 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: RocksDB version: 7.9.2 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Git sha 0 2024-04-22T06:29:30.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Compile date 2024-04-21 19:54:05 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: DB SUMMARY 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: DB Session ID: IDWU7581KAFDENBFBFZC 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: CURRENT file: CURRENT 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.error_if_exists: 0 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.create_if_missing: 0 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.paranoid_checks: 1 2024-04-22T06:29:30.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.env: 0x56247aadacc0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.fs: PosixFileSystem 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.info_log: 0x56247bfa3be0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.statistics: (nil) 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.use_fsync: 0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_log_file_size: 0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T06:29:30.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_fallocate: 1 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.use_direct_reads: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.db_log_dir: 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.wal_dir: 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T06:29:30.744 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.write_buffer_manager: 0x56247bf370e0 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T06:29:30.745 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.unordered_write: 0 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.row_cache: None 2024-04-22T06:29:30.746 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.wal_filter: None 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.two_write_queues: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.wal_compression: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.atomic_flush: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.log_readahead_size: 0 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T06:29:30.748 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_background_jobs: 2 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_background_compactions: -1 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_subcompactions: 1 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T06:29:30.749 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_open_files: -1 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_background_flushes: -1 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Compression algorithms supported: 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kZSTD supported: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kXpressCompression supported: 0 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T06:29:30.750 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kZlibCompression supported: 1 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kSnappyCompression supported: 1 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kLZ4Compression supported: 1 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: kBZip2Compression supported: 0 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.merge_operator: 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_filter: None 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56247bfa3dc0) 2024-04-22T06:29:30.751 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_type: 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_index_type: 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_shortening: 1 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: checksum: 4 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: no_block_cache: 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache: 0x56247bf691f0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_options: 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: capacity : 536870912 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_shard_bits : 4 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: strict_capacity_limit : 0 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-04-22T06:29:30.752 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_cache_compressed: (nil) 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: persistent_cache: (nil) 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size: 4096 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_size_deviation: 10 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_restart_interval: 16 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: index_block_restart_interval: 1 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: metadata_block_size: 4096 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: partition_filters: 0 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: use_delta_encoding: 1 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: filter_policy: bloomfilter 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: whole_key_filtering: 1 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: verify_compression: 0 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: format_version: 5 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_index_compression: 1 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: block_align: 0 2024-04-22T06:29:30.753 INFO:journalctl@ceph.mon.c.smithi177.stdout: max_auto_readahead_size: 262144 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout: prepopulate_block_cache: 0 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout: initial_auto_readahead_size: 8192 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression: NoCompression 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.num_levels: 7 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T06:29:30.754 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-22T06:29:30.755 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T06:29:30.756 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.arena_block_size: 1048576 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T06:29:30.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.inplace_update_support: 0 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.bloom_locality: 0 2024-04-22T06:29:30.758 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.max_successive_merges: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.ttl: 2592000 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enable_blob_files: false 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.min_blob_size: 0 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T06:29:30.759 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.blob_file_starting_level: 0 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e51ba075-5617-47c4-9c49-65d77e205db5 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767370411187, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767370411595, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713767370, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e51ba075-5617-47c4-9c49-65d77e205db5", "db_session_id": "IDWU7581KAFDENBFBFZC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713767370411673, "job": 1, "event": "recovery_finished"} 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56247c07e000 2024-04-22T06:29:30.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: DB pointer 0x56247c068000 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** DB Stats ** 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:29:30.761 INFO:journalctl@ceph.mon.c.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** Compaction Stats [default] ** 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T06:29:30.762 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache BinnedLRUCache@0x56247bf691f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: using public_addr v2:172.21.15.177:0/0 -> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: starting mon.c rank -1 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:30.763 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(???) e0 preinit fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).mds e1 new map 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: e1 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: legacy client fscid: -1 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout: No filesystems configured 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T06:29:30.764 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:29:30.765 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Deploying cephadm binary to smithi177 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Added host smithi177 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.766 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3683906140' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3683906140' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.767 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:30.768 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Saving service mon spec with placement smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c;count:3 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: Deploying daemon mon.c on smithi177 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/1057787542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 ceph-mon[31346]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 bash[31327]: f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 2024-04-22T06:29:30.769 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:30 smithi177 systemd[1]: Started Ceph mon.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:29:31.010 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:29:31.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:29:31.243 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:29:35.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1880947495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 msgr2=0x7ffa44005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1880947495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa44005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c101360 tx=0x7ffa5000feb0 comp rx=0 tx=0).stop 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1880947495 shutdown_connections 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1880947495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa44005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1880947495 >> 172.21.15.177:0/1880947495 conn(0x7ffa5c0fcf20 msgr2=0x7ffa5c0ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1880947495 shutdown_connections 2024-04-22T06:29:35.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.756+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1880947495 wait complete. 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 Processor -- start 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 -- start start 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa5c198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c109380 con 0x7ffa440035c0 2024-04-22T06:29:35.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.757+0000 7ffa60b3d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c109200 con 0x7ffa5c199310 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:38038/0 (socket says 172.21.15.177:38038) 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 -- 172.21.15.177:0/1508641403 learned_addr learned my addr 172.21.15.177:0/1508641403 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa5a59c700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa5c198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 -- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 msgr2=0x7ffa5c199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 -- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 msgr2=0x7ffa5c199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa59d9b700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa5a59c700 1 -- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 msgr2=0x7ffa5c199790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa5a59c700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa5a59c700 1 -- 172.21.15.177:0/1508641403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c1a1e40 con 0x7ffa440035c0 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.758+0000 7ffa5a59c700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa5c198dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffa50010010 tx=0x7ffa50005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.759+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ffa50004aa0 con 0x7ffa440035c0 2024-04-22T06:29:35.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.759+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5c1a2010 con 0x7ffa440035c0 2024-04-22T06:29:35.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.759+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa50004030 con 0x7ffa440035c0 2024-04-22T06:29:35.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.759+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5c1a25a0 con 0x7ffa440035c0 2024-04-22T06:29:35.760 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.759+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ffa50011af0 con 0x7ffa440035c0 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: mon.a calling monitor election 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: mon.c calling monitor election 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.760 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: monmap epoch 2 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: last_changed 2024-04-22T06:29:30.440185+0000 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: min_mon_release 19 (squid) 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: election_strategy: 1 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: fsmap 2024-04-22T06:29:35.761 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: mgrmap e13: x(active, since 50s) 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: overall HEALTH_OK 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:35.762 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.760+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7ffa50023070 con 0x7ffa440035c0 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.760+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5c102b70 con 0x7ffa440035c0 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.760+0000 7ffa4b7fe700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 0x7ffa3c03b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.760+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffa50020050 con 0x7ffa440035c0 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.761+0000 7ffa59d9b700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 0x7ffa3c03b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:35.763 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.761+0000 7ffa59d9b700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 0x7ffa3c03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009180 tx=0x7ffa4c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:35.765 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:35.766+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa50028020 con 0x7ffa440035c0 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: mon.a calling monitor election 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: mon.c calling monitor election 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: monmap epoch 2 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: last_changed 2024-04-22T06:29:30.440185+0000 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: min_mon_release 19 (squid) 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: election_strategy: 1 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:29:35.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: fsmap 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: mgrmap e13: x(active, since 50s) 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: overall HEALTH_OK 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:35.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:36.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.048+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ffa5c0640a0 con 0x7ffa440035c0 2024-04-22T06:29:36.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.049+0000 7ffa4b7fe700 1 -- 172.21.15.177:0/1508641403 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7ffa5001a390 con 0x7ffa440035c0 2024-04-22T06:29:36.048 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:29:36.048 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","modified":"2024-04-22T06:29:30.440185Z","created":"2024-04-22T06:27:38.293503Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T06:29:36.050 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 msgr2=0x7ffa3c03b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:36.050 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 0x7ffa3c03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009180 tx=0x7ffa4c008040 comp rx=0 tx=0).stop 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 msgr2=0x7ffa5c198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa5c198dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ffa50010010 tx=0x7ffa50005020 comp rx=0 tx=0).stop 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 shutdown_connections 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffa3c0393f0 0x7ffa3c03b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffa440035c0 0x7ffa5c198dd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 --2- 172.21.15.177:0/1508641403 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffa5c199310 0x7ffa5c199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 >> 172.21.15.177:0/1508641403 conn(0x7ffa5c0fcf20 msgr2=0x7ffa5c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 shutdown_connections 2024-04-22T06:29:36.051 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:36.051+0000 7ffa60b3d700 1 -- 172.21.15.177:0/1508641403 wait complete. 2024-04-22T06:29:36.052 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-04-22T06:29:36.626 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:36 smithi177 ceph-mon[31346]: Deploying daemon mon.b on smithi112 2024-04-22T06:29:36.626 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:36 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/1508641403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:36.626 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:36 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:36 smithi105 ceph-mon[27916]: Deploying daemon mon.b on smithi112 2024-04-22T06:29:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:36 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/1508641403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:36 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:37.656 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T06:29:37.656 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T06:29:37.899 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:29:37.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:37 smithi105 ceph-mon[27916]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:37 smithi177 ceph-mon[31346]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:44.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: mon.a calling monitor election 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: mon.c calling monitor election 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: mon.b calling monitor election 2024-04-22T06:29:44.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: monmap epoch 3 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: last_changed 2024-04-22T06:29:39.387606+0000 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: min_mon_release 19 (squid) 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: election_strategy: 1 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-22T06:29:44.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: fsmap 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: mgrmap e13: x(active, since 59s) 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: overall HEALTH_OK 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.743 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:29:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: mon.a calling monitor election 2024-04-22T06:29:44.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: mon.c calling monitor election 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: mon.b calling monitor election 2024-04-22T06:29:44.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: monmap epoch 3 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:44.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: last_changed 2024-04-22T06:29:39.387606+0000 2024-04-22T06:29:44.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: created 2024-04-22T06:27:38.293503+0000 2024-04-22T06:29:44.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: min_mon_release 19 (squid) 2024-04-22T06:29:44.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: election_strategy: 1 2024-04-22T06:29:44.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: 0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-22T06:29:44.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: 1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: fsmap 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: mgrmap e13: x(active, since 59s) 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: overall HEALTH_OK 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.923 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.924 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:44.924 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 -- 172.21.15.177:0/1576972447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6380084b0 msgr2=0x7fa6380088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 --2- 172.21.15.177:0/1576972447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6380084b0 0x7fa6380088d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa64c102510 tx=0x7fa64800fe90 comp rx=0 tx=0).stop 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 -- 172.21.15.177:0/1576972447 shutdown_connections 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 --2- 172.21.15.177:0/1576972447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6380084b0 0x7fa6380088d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 -- 172.21.15.177:0/1576972447 >> 172.21.15.177:0/1576972447 conn(0x7fa64c0fcf80 msgr2=0x7fa64c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 -- 172.21.15.177:0/1576972447 shutdown_connections 2024-04-22T06:29:45.400 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.398+0000 7fa653566700 1 -- 172.21.15.177:0/1576972447 wait complete. 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 Processor -- start 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 -- start start 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 0x7fa64c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c1082f0 con 0x7fa64c19a9e0 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c06bb60 con 0x7fa6380084b0 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa653566700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c108470 con 0x7fa64c19f2b0 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa651b03700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.399+0000 7fa651b03700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:58310/0 (socket says 172.21.15.177:58310) 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651b03700 1 -- 172.21.15.177:0/3428385528 learned_addr learned my addr 172.21.15.177:0/3428385528 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651302700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:45.401 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651b03700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 msgr2=0x7fa64c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 0x7fa64c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651b03700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 msgr2=0x7fa64c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651b03700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651302700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 msgr2=0x7fa64c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651302700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 msgr2=0x7fa64c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651302700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651302700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa651b03700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 msgr2=0x7fa64c19f760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 msgr2=0x7fa64c19a4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.402 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 -- 172.21.15.177:0/3428385528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c1a7020 con 0x7fa64c19a9e0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa650b01700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 0x7fa64c19ed70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00b550 tx=0x7fa63c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c004230 con 0x7fa64c19a9e0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64c1a7250 con 0x7fa64c19a9e0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.400+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa63c0043d0 con 0x7fa64c19a9e0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.401+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c011b90 con 0x7fa64c19a9e0 2024-04-22T06:29:45.403 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.401+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64c1a77e0 con 0x7fa64c19a9e0 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.401+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fa63c01a400 con 0x7fa64c19a9e0 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.402+0000 7fa6427fc700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 0x7fa63003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.402+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa63c04c390 con 0x7fa64c19a9e0 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.402+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c0640a0 con 0x7fa64c19a9e0 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.402+0000 7fa651302700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 0x7fa63003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:45.404 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.402+0000 7fa651302700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 0x7fa63003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa648005b40 tx=0x7fa648005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:45.406 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.408+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa63c020500 con 0x7fa64c19a9e0 2024-04-22T06:29:45.638 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:45 smithi177 ceph-mon[31346]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:45.639 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:45.639 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:45.639 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:45.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:45 smithi112 ceph-mon[31236]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:45.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:45.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:45.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:45.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.715+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa64c1a7b00 con 0x7fa64c19a9e0 2024-04-22T06:29:45.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.715+0000 7fa6427fc700 1 -- 172.21.15.177:0/3428385528 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7fa63c0202a0 con 0x7fa64c19a9e0 2024-04-22T06:29:45.715 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-04-22T06:29:45.715 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","modified":"2024-04-22T06:29:39.387606Z","created":"2024-04-22T06:27:38.293503Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 msgr2=0x7fa63003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 0x7fa63003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa648005b40 tx=0x7fa648005a90 comp rx=0 tx=0).stop 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 msgr2=0x7fa64c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 0x7fa64c19ed70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00b550 tx=0x7fa63c00b910 comp rx=0 tx=0).stop 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 shutdown_connections 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.717+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa630039ae0 0x7fa63003bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6380084b0 0x7fa64c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.717 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa64c19f2b0 0x7fa64c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.718 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 --2- 172.21.15.177:0/3428385528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa64c19a9e0 0x7fa64c19ed70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:45.718 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 >> 172.21.15.177:0/3428385528 conn(0x7fa64c0fcf80 msgr2=0x7fa64c102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:45.718 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 shutdown_connections 2024-04-22T06:29:45.718 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:45.718+0000 7fa653566700 1 -- 172.21.15.177:0/3428385528 wait complete. 2024-04-22T06:29:45.718 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-04-22T06:29:45.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:45 smithi105 ceph-mon[27916]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:45.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:45.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:45.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:29:46.471 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T06:29:46.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T06:29:46.709 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:46.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/3428385528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:46.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:46.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:46.831 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/3428385528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.832 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:46.833 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:46.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/3428385528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T06:29:46.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:46.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:46.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:46.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:47.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:29:47.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:29:47.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:29:47.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.857 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.857 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:47.857 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:29:47.857 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:47 smithi105 ceph-mon[27916]: Reconfiguring daemon mon.a on smithi105 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:29:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:47 smithi112 ceph-mon[31236]: Reconfiguring daemon mon.a on smithi105 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:29:47.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:47.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Reconfiguring mon.a (unknown last config time)... 2024-04-22T06:29:47.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:47 smithi177 ceph-mon[31346]: Reconfiguring daemon mon.a on smithi105 2024-04-22T06:29:48.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 -- 172.21.15.105:0/3348487189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 msgr2=0x7f1138101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:48.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 --2- 172.21.15.105:0/3348487189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f1138101970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1128009910 tx=0x7f1128009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:48.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 -- 172.21.15.105:0/3348487189 shutdown_connections 2024-04-22T06:29:48.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 --2- 172.21.15.105:0/3348487189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f1138101970 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 -- 172.21.15.105:0/3348487189 >> 172.21.15.105:0/3348487189 conn(0x7f11380fce40 msgr2=0x7f11380ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:48.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 -- 172.21.15.105:0/3348487189 shutdown_connections 2024-04-22T06:29:48.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.361+0000 7f113de0f700 1 -- 172.21.15.105:0/3348487189 wait complete. 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 Processor -- start 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 -- start start 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 0x7f113819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f113819cef0 0x7f113819d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f113810b810 con 0x7f1138101590 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f113810b690 con 0x7f1138198620 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f113de0f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f113810b990 con 0x7f113819cef0 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f11377fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f11377fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32934/0 (socket says 172.21.15.105:32934) 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.362+0000 7f11377fe700 1 -- 172.21.15.105:0/521886330 learned_addr learned my addr 172.21.15.105:0/521886330 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:48.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f1136ffd700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 0x7f113819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 -- 172.21.15.105:0/521886330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f113819cef0 msgr2=0x7f113819d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f113819cef0 0x7f113819d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 -- 172.21.15.105:0/521886330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 msgr2=0x7f113819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 0x7f113819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 -- 172.21.15.105:0/521886330 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11381a4c60 con 0x7f1138101590 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f1136ffd700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 0x7f113819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f11377fe700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1128005b40 tx=0x7f1128005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128026070 con 0x7f1138101590 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f112800b6f0 con 0x7f1138101590 2024-04-22T06:29:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11280038e0 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11381a4e90 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.363+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11381a52f0 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.364+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f1128029070 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.364+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11380640a0 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.364+0000 7f1134ff9700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 0x7f112003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.364+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f112804d640 con 0x7f1138101590 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.364+0000 7f1136ffd700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 0x7f112003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.365+0000 7f1136ffd700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 0x7f112003bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f112c0079b0 tx=0x7f112c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:48.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.369+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1128013070 con 0x7f1138101590 2024-04-22T06:29:48.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.578+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f11381a56b0 con 0x7f1138101590 2024-04-22T06:29:48.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.578+0000 7f1134ff9700 1 -- 172.21.15.105:0/521886330 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f1128030090 con 0x7f1138101590 2024-04-22T06:29:48.577 INFO:teuthology.orchestra.run.smithi105.stdout:# minimal ceph.conf for 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:48.578 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-04-22T06:29:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:29:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: mon_host = [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-04-22T06:29:48.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 msgr2=0x7f112003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:48.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 0x7f112003bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f112c0079b0 tx=0x7f112c008040 comp rx=0 tx=0).stop 2024-04-22T06:29:48.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 msgr2=0x7f11381980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:48.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1128005b40 tx=0x7f1128005030 comp rx=0 tx=0).stop 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 shutdown_connections 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1120039ae0 0x7f112003bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f113819cef0 0x7f113819d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1138101590 0x7f11381980e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 --2- 172.21.15.105:0/521886330 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1138198620 0x7f113819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.580+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 >> 172.21.15.105:0/521886330 conn(0x7f11380fce40 msgr2=0x7f113806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.581+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 shutdown_connections 2024-04-22T06:29:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:48.581+0000 7f113de0f700 1 -- 172.21.15.105:0/521886330 wait complete. 2024-04-22T06:29:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/521886330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:48.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/521886330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:48.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/521886330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:49.337 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T06:29:49.337 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:29:49.338 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:29:49.365 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:29:49.365 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:49.429 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:29:49.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:29:49.460 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:29:49.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:49.529 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:29:49.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T06:29:49.560 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:29:49.560 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:29:49.627 INFO:tasks.cephadm:Adding mgr.x on smithi105 2024-04-22T06:29:49.627 INFO:tasks.cephadm:Adding mgr.y on smithi112 2024-04-22T06:29:49.628 INFO:tasks.cephadm:Adding mgr.z on smithi177 2024-04-22T06:29:49.628 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi105=x;smithi112=y;smithi177=z' 2024-04-22T06:29:49.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:49 smithi112 ceph-mon[31236]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:49.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:49 smithi105 ceph-mon[27916]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:49.902 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:29:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:49 smithi177 ceph-mon[31346]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:50.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: Reconfiguring daemon mon.b on smithi112 2024-04-22T06:29:50.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:50 smithi177 ceph-mon[31346]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: Reconfiguring daemon mon.b on smithi112 2024-04-22T06:29:51.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:50 smithi105 ceph-mon[27916]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:51.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 -- 172.21.15.177:0/2682005731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4310102ae0 msgr2=0x7f4310102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:51.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 --2- 172.21.15.177:0/2682005731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4310102ae0 0x7f4310102f00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f430c009910 tx=0x7f430c009c20 comp rx=0 tx=0).stop 2024-04-22T06:29:51.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 -- 172.21.15.177:0/2682005731 shutdown_connections 2024-04-22T06:29:51.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 --2- 172.21.15.177:0/2682005731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4310102ae0 0x7f4310102f00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 -- 172.21.15.177:0/2682005731 >> 172.21.15.177:0/2682005731 conn(0x7f43100fc1f0 msgr2=0x7f43100fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 -- 172.21.15.177:0/2682005731 shutdown_connections 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.198+0000 7f43179d3700 1 -- 172.21.15.177:0/2682005731 wait complete. 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 Processor -- start 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 -- start start 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4310102ae0 0x7f4310199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f431019a3b0 0x7f431019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f431010a650 con 0x7f431019a3b0 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f431010a4d0 con 0x7f4310102ae0 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f43179d3700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f431010a7d0 con 0x7f431019a830 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f4315f70700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f4315f70700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:58326/0 (socket says 172.21.15.177:58326) 2024-04-22T06:29:51.200 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f4315f70700 1 -- 172.21.15.177:0/931829136 learned_addr learned my addr 172.21.15.177:0/931829136 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f4314f6e700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f431019a3b0 0x7f431019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.199+0000 7f431576f700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4310102ae0 0x7f4310199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 -- 172.21.15.177:0/931829136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4310102ae0 msgr2=0x7f4310199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4310102ae0 0x7f4310199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 -- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f431019a3b0 msgr2=0x7f431019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f431019a3b0 0x7f431019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 -- 172.21.15.177:0/931829136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43101a2280 con 0x7f431019a830 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f4315f70700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f430800ba50 tx=0x7f430800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308019070 con 0x7f431019a830 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430800cde0 con 0x7f431019a830 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43101a2570 con 0x7f431019a830 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308007350 con 0x7f431019a830 2024-04-22T06:29:51.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.200+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43101a2ae0 con 0x7f431019a830 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: Reconfiguring mon.b (monmap changed)... 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: Reconfiguring daemon mon.b on smithi112 2024-04-22T06:29:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:50 smithi112 ceph-mon[31236]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:51.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.201+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f430801c070 con 0x7f431019a830 2024-04-22T06:29:51.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.201+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43100640a0 con 0x7f431019a830 2024-04-22T06:29:51.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.201+0000 7f43067fc700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 0x7f42fc03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:51.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.201+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f430804be10 con 0x7f431019a830 2024-04-22T06:29:51.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.201+0000 7f431576f700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 0x7f42fc03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:51.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.202+0000 7f431576f700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 0x7f42fc03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f430c005190 tx=0x7f430c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:51.205 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.206+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f430800f160 con 0x7f431019a830 2024-04-22T06:29:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.424+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi105=x;smithi112=y;smithi177=z", "target": ["mon-mgr", ""]}) -- 0x7f4310062fd0 con 0x7f42fc039ae0 2024-04-22T06:29:51.429 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.430+0000 7f43067fc700 1 -- 172.21.15.177:0/931829136 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4310062fd0 con 0x7f42fc039ae0 2024-04-22T06:29:51.430 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.432+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 msgr2=0x7f42fc03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.432+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 0x7f42fc03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f430c005190 tx=0x7f430c01a040 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.432+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 msgr2=0x7f43101a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.432+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f430800ba50 tx=0x7f430800bd60 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 shutdown_connections 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42fc039ae0 0x7f42fc03bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4310102ae0 0x7f4310199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f431019a830 0x7f43101a19b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 --2- 172.21.15.177:0/931829136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f431019a3b0 0x7f431019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 >> 172.21.15.177:0/931829136 conn(0x7f43100fc1f0 msgr2=0x7f43100fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 shutdown_connections 2024-04-22T06:29:51.432 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:29:51.433+0000 7f43179d3700 1 -- 172.21.15.177:0/931829136 wait complete. 2024-04-22T06:29:52.000 DEBUG:teuthology.orchestra.run.smithi112:mgr.y> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T06:29:52.002 DEBUG:teuthology.orchestra.run.smithi177:mgr.z> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.z.service 2024-04-22T06:29:52.005 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T06:29:52.005 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:29:52.005 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:29:52.024 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:29:52.025 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-22 06:28:57.174429203 +0000 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-22 06:23:43.838334618 +0000 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-22 06:23:43.838334618 +0000 2024-04-22T06:29:52.083 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-22 06:23:43.838334618 +0000 2024-04-22T06:29:52.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:29:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-22T06:29:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-22T06:29:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000218696 s, 2.3 MB/s 2024-04-22T06:29:52.151 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:29:52.210 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-04-22T06:29:52.266 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:29:52.266 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-22 06:28:57.219428323 +0000 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-22 06:23:44.175327974 +0000 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-22 06:23:44.175327974 +0000 2024-04-22T06:29:52.267 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-22 06:23:44.175327974 +0000 2024-04-22T06:29:52.267 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:29:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-22T06:29:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-22T06:29:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000248012 s, 2.1 MB/s 2024-04-22T06:29:52.333 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:29:52.390 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: Reconfiguring daemon mon.c on smithi177 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi105=x;smithi112=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: Saving service mgr spec with placement smithi105=x;smithi112=y;smithi177=z;count:3 2024-04-22T06:29:52.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:52 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-22 06:28:57.255427618 +0000 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-22 06:23:44.516321251 +0000 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-22 06:23:44.516321251 +0000 2024-04-22T06:29:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-22 06:23:44.516321251 +0000 2024-04-22T06:29:52.435 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: Reconfiguring daemon mon.c on smithi177 2024-04-22T06:29:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi105=x;smithi112=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:52.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: Saving service mgr spec with placement smithi105=x;smithi112=y;smithi177=z;count:3 2024-04-22T06:29:52.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:52 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.498 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-22T06:29:52.499 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-22T06:29:52.499 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000236597 s, 2.2 MB/s 2024-04-22T06:29:52.500 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:29:52.556 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-22 06:28:57.287426992 +0000 2024-04-22T06:29:52.612 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-22 06:23:44.852314627 +0000 2024-04-22T06:29:52.613 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-22 06:23:44.852314627 +0000 2024-04-22T06:29:52.613 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-22 06:23:44.852314627 +0000 2024-04-22T06:29:52.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:29:52.675 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-22T06:29:52.675 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-22T06:29:52.675 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000234532 s, 2.2 MB/s 2024-04-22T06:29:52.676 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:29:52.733 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-04-22T06:29:52.733 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:29:52.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: Reconfiguring mon.c (monmap changed)... 2024-04-22T06:29:52.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: Reconfiguring daemon mon.c on smithi177 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi105=x;smithi112=y;smithi177=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: Saving service mgr spec with placement smithi105=x;smithi112=y;smithi177=z;count:3 2024-04-22T06:29:52.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:52 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:52.752 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:29:52.752 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-22 06:29:38.849472760 +0000 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-22 06:24:05.567292578 +0000 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-22 06:24:05.567292578 +0000 2024-04-22T06:29:52.810 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-22 06:24:05.567292578 +0000 2024-04-22T06:29:52.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:29:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-22T06:29:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-22T06:29:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000288991 s, 1.8 MB/s 2024-04-22T06:29:52.877 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:29:52.935 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-04-22T06:29:52.991 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:29:52.991 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-22 06:29:38.849472760 +0000 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-22 06:24:05.928283154 +0000 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-22 06:24:05.928283154 +0000 2024-04-22T06:29:52.992 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-22 06:24:05.928283154 +0000 2024-04-22T06:29:52.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:29:53.056 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-22T06:29:53.057 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-22T06:29:53.057 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.00021415 s, 2.4 MB/s 2024-04-22T06:29:53.058 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:29:53.115 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-04-22T06:29:53.171 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-22 06:29:38.849472760 +0000 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-22 06:24:06.262274435 +0000 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-22 06:24:06.262274435 +0000 2024-04-22T06:29:53.172 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-22 06:24:06.262274435 +0000 2024-04-22T06:29:53.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:29:53.236 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-22T06:29:53.236 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-22T06:29:53.236 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000196237 s, 2.6 MB/s 2024-04-22T06:29:53.238 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:29:53.294 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-04-22 06:29:38.850472734 +0000 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-04-22 06:24:06.593265794 +0000 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-04-22 06:24:06.593265794 +0000 2024-04-22T06:29:53.350 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-04-22 06:24:06.593265794 +0000 2024-04-22T06:29:53.351 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:29:53.414 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-04-22T06:29:53.414 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-04-22T06:29:53.414 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000191081 s, 2.7 MB/s 2024-04-22T06:29:53.414 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:29:53.470 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-04-22T06:29:53.470 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:29:53.489 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:29:53.489 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-04-22T06:29:53.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:53 smithi177 ceph-mon[31346]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-22 06:29:29.319161009 +0000 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-22 06:24:05.147190567 +0000 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-22 06:24:05.147190567 +0000 2024-04-22T06:29:53.506 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-22 06:24:05.147190567 +0000 2024-04-22T06:29:53.507 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:29:53.571 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-22T06:29:53.571 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-22T06:29:53.571 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000299494 s, 1.7 MB/s 2024-04-22T06:29:53.572 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:29:53.630 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-04-22T06:29:53.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:53 smithi105 ceph-mon[27916]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-22 06:29:29.320160977 +0000 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-22 06:24:05.480179748 +0000 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-22 06:24:05.480179748 +0000 2024-04-22T06:29:53.686 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-22 06:24:05.480179748 +0000 2024-04-22T06:29:53.687 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:29:53.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:53 smithi112 ceph-mon[31236]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:53.750 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-22T06:29:53.750 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-22T06:29:53.750 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000247773 s, 2.1 MB/s 2024-04-22T06:29:53.751 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:29:53.808 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-22 06:29:29.320160977 +0000 2024-04-22T06:29:53.864 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-22 06:24:05.824168572 +0000 2024-04-22T06:29:53.865 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-22 06:24:05.824168572 +0000 2024-04-22T06:29:53.865 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-22 06:24:05.824168572 +0000 2024-04-22T06:29:53.865 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:29:53.927 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-22T06:29:53.927 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-22T06:29:53.927 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000215381 s, 2.4 MB/s 2024-04-22T06:29:53.928 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:29:53.983 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-04-22 06:29:29.321160944 +0000 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-04-22 06:24:06.162157590 +0000 2024-04-22T06:29:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-04-22 06:24:06.162157590 +0000 2024-04-22T06:29:54.040 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-04-22 06:24:06.162157590 +0000 2024-04-22T06:29:54.040 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:29:54.102 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-04-22T06:29:54.102 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-04-22T06:29:54.102 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000177117 s, 2.9 MB/s 2024-04-22T06:29:54.103 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:29:54.159 INFO:tasks.cephadm:Deploying osd.0 on smithi105 with /dev/vg_nvme/lv_4... 2024-04-22T06:29:54.159 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:29:54.386 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:54.508 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:54.509 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:54 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:54.518 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:54.519 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:54 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:29:54.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:54.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:54.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:54.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:54.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:54 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:55.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:55 smithi112 ceph-mon[31236]: Deploying daemon mgr.z on smithi177 2024-04-22T06:29:55.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:55 smithi112 ceph-mon[31236]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:55.617 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:55 smithi105 ceph-mon[27916]: Deploying daemon mgr.z on smithi177 2024-04-22T06:29:55.617 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:55 smithi105 ceph-mon[27916]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:55.631 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:55 smithi177 ceph-mon[31346]: Deploying daemon mgr.z on smithi177 2024-04-22T06:29:55.631 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:55 smithi177 ceph-mon[31346]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:55.990 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:55 smithi177 systemd[1]: Starting Ceph mgr.z for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114297 s, 917 MB/s 2024-04-22T06:29:56.313 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 podman[33193]: 2024-04-22 06:29:56.047829846 +0000 UTC m=+0.113195318 container create a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 podman[33193]: 2024-04-22 06:29:55.952299963 +0000 UTC m=+0.017665470 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 podman[33193]: 2024-04-22 06:29:56.199979547 +0000 UTC m=+0.265345031 container init a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 podman[33193]: 2024-04-22 06:29:56.202433701 +0000 UTC m=+0.267799185 container start a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 ceph-mgr[33214]: -- 172.21.15.177:0/1777757271 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce238f65a0 con 0x55ce238ca800 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 bash[33193]: a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 systemd[1]: Started Ceph mgr.z for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:29:56.361 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:56.361+0000 7ff18155d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:29:56.646 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:56.545+0000 7ff18155d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:29:56.646 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:56.646+0000 7ff18155d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:29:56.982 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-04-22T06:29:57.211 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:29:57.240 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:56.949+0000 7ff18155d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:29:57.240 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:57.032+0000 7ff18155d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: Deploying daemon mgr.y on smithi112 2024-04-22T06:29:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:57 smithi112 ceph-mon[31236]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:57.608 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.608 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: Deploying daemon mgr.y on smithi112 2024-04-22T06:29:57.609 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:57 smithi105 ceph-mon[27916]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:57.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:57.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:29:57.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T06:29:57.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:29:57.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:57.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: Deploying daemon mgr.y on smithi112 2024-04-22T06:29:57.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:57 smithi177 ceph-mon[31346]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:58.459 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 systemd[1]: Starting Ceph mgr.y for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:29:58.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:58.215+0000 7ff18155d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:29:58.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:58.381+0000 7ff18155d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:29:58.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 -- 172.21.15.105:0/3441657139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a4103540 msgr2=0x7fb6a4109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:58.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/3441657139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a4103540 0x7fb6a4109df0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00b0d0 tx=0x7fb69c00b3e0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 -- 172.21.15.105:0/3441657139 shutdown_connections 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/3441657139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a41019a0 0x7fb6a4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/3441657139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a4103540 0x7fb6a4109df0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/3441657139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4102ba0 0x7fb6a4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 -- 172.21.15.105:0/3441657139 >> 172.21.15.105:0/3441657139 conn(0x7fb6a40fcf40 msgr2=0x7fb6a40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.644+0000 7fb6a96d4700 1 -- 172.21.15.105:0/3441657139 shutdown_connections 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/3441657139 wait complete. 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 Processor -- start 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 -- start start 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a4102ba0 0x7fb6a4103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4103540 0x7fb6a41044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a410d420 con 0x7fb6a41019a0 2024-04-22T06:29:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a410d2a0 con 0x7fb6a4102ba0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.645+0000 7fb6a96d4700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a410d5a0 con 0x7fb6a4103540 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43694/0 (socket says 172.21.15.105:43694) 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 -- 172.21.15.105:0/1867761889 learned_addr learned my addr 172.21.15.105:0/1867761889 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a37fe700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4103540 0x7fb6a41044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a27fc700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a4102ba0 0x7fb6a4103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 -- 172.21.15.105:0/1867761889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4103540 msgr2=0x7fb6a41044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4103540 0x7fb6a41044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 -- 172.21.15.105:0/1867761889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a4102ba0 msgr2=0x7fb6a4103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a4102ba0 0x7fb6a4103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 -- 172.21.15.105:0/1867761889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a4104d90 con 0x7fb6a41019a0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a2ffd700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb69400d980 tx=0x7fb69400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694017030 con 0x7fb6a41019a0 2024-04-22T06:29:58.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a41a2d20 con 0x7fb6a41019a0 2024-04-22T06:29:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6940044d0 con 0x7fb6a41019a0 2024-04-22T06:29:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694003970 con 0x7fb6a41019a0 2024-04-22T06:29:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.646+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a41a3170 con 0x7fb6a41019a0 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.647+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fb69401a070 con 0x7fb6a41019a0 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.647+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a41a3400 con 0x7fb6a41019a0 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.647+0000 7fb68bfff700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 0x7fb68c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.647+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb69404c410 con 0x7fb6a41019a0 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.647+0000 7fb6a27fc700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 0x7fb68c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:29:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.648+0000 7fb6a27fc700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 0x7fb68c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb698007be0 tx=0x7fb6980073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:29:58.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.652+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb694009ba0 con 0x7fb6a41019a0 2024-04-22T06:29:58.797 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 podman[32621]: 2024-04-22 06:29:58.448346714 +0000 UTC m=+0.017754503 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:29:58.797 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 podman[32621]: 2024-04-22 06:29:58.586849069 +0000 UTC m=+0.156256843 container create ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-22T06:29:58.797 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 podman[32621]: 2024-04-22 06:29:58.772323833 +0000 UTC m=+0.341731606 container init ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:29:58.798 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 podman[32621]: 2024-04-22 06:29:58.774791631 +0000 UTC m=+0.344199406 container start ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-22T06:29:58.798 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 ceph-mgr[32641]: -- 172.21.15.112:0/1178548766 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592d0eec5a0 con 0x5592d0ec0800 2024-04-22T06:29:58.835 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:58.552+0000 7ff18155d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:29:58.835 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:58.836+0000 7ff18155d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:29:58.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:29:58.860+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb6a40ff110 con 0x7fb68c039890 2024-04-22T06:29:59.123 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 bash[32621]: ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa 2024-04-22T06:29:59.123 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 systemd[1]: Started Ceph mgr.y for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:29:59.123 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:29:58.938+0000 7f086411c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:29:59.124 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:29:59.124+0000 7f086411c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:29:59.133 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:58.927+0000 7ff18155d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:29:59.133 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:59.134+0000 7ff18155d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:29:59.459 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:29:59.232+0000 7f086411c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:29:59.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:59.331+0000 7ff18155d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.801 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.802 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:59.802 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:29:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:59.881 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.909 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:29:59.552+0000 7f086411c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:29:59.647+0000 7f086411c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:29:59.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:29:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.135 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:29:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:29:59.964+0000 7ff18155d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:30:00.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: overall HEALTH_OK 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:30:00.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:00.696 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:00.397+0000 7ff18155d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:30:00.696 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:00.495+0000 7ff18155d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:30:00.696 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:00.697+0000 7ff18155d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:30:00.696 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: overall HEALTH_OK 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:30:00.697 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: overall HEALTH_OK 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T06:30:00.869 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:01.199 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:00 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:00.868+0000 7f086411c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:30:01.199 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.034+0000 7f086411c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:30:01.199 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.199+0000 7f086411c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:30:01.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.217+0000 7ff18155d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:30:01.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.308+0000 7ff18155d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:30:01.769 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.480+0000 7f086411c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:30:01.770 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.571+0000 7f086411c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:30:01.770 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.770+0000 7f086411c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:30:01.770 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:01.770 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:30:01.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: Reconfiguring daemon mgr.x on smithi105 2024-04-22T06:30:01.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]: dispatch 2024-04-22T06:30:01.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]': finished 2024-04-22T06:30:01.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:30:01.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: Reconfiguring daemon mgr.x on smithi105 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]: dispatch 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]': finished 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:01.886 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.548+0000 7ff18155d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:30:01.887 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.680+0000 7ff18155d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:30:01.887 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.773+0000 7ff18155d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:30:01.887 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:01.886+0000 7ff18155d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: Reconfiguring mgr.x (unknown last config time)... 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: Reconfiguring daemon mgr.x on smithi105 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]: dispatch 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3645801064' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a4d5ec0-d76f-4b44-9fcc-01b111b51a17"}]': finished 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T06:30:01.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:02.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:01 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:01.958+0000 7f086411c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:30:02.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:02.188+0000 7ff18155d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:30:02.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:02 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1150525276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:02.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:02 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1150525276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:02.953 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:02 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:02.539+0000 7f086411c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:30:02.953 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:02 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:02.953+0000 7f086411c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:30:02.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:02 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1150525276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:02.990 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:02.784+0000 7ff18155d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:30:03.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:03.049+0000 7f086411c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:30:03.490 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:30:02.998+0000 7ff18155d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:30:03.614 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:03.249+0000 7f086411c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:30:03.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:03.626+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 14) ==== 92603+0+0 (secure 0 0 0) 0x7fb69401ccf0 con 0x7fb6a41019a0 2024-04-22T06:30:03.865 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:03.774+0000 7f086411c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:30:03.865 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:03.865+0000 7f086411c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: Standby manager daemon z started 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:30:03.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:03 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: Standby manager daemon z started 2024-04-22T06:30:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:30:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:03.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:30:03.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:03 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: Standby manager daemon z started 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:30:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:03 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/2206125748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:04.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:04.105+0000 7f086411c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:30:04.629 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:04.236+0000 7f086411c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:30:04.629 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:04.330+0000 7f086411c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:30:04.629 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:04.446+0000 7f086411c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:30:04.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:04 smithi105 ceph-mon[27916]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T06:30:04.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:04 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:30:04.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-mon[31236]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T06:30:04.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:30:04.959 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:04 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:04.751+0000 7f086411c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:30:04.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:04 smithi177 ceph-mon[31346]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-22T06:30:04.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:04 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:30:05.630 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:05.343+0000 7f086411c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:30:05.630 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:30:05.555+0000 7f086411c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:30:05.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:05.640+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fb69401a360 con 0x7fb6a41019a0 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: Standby manager daemon y started 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:30:05.728 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:05 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: Standby manager daemon y started 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:30:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:05 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: Standby manager daemon y started 2024-04-22T06:30:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:30:05.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:30:05.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:30:05.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:05 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/3172937835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:30:06.903 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:06 smithi112 ceph-mon[31236]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T06:30:06.903 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:30:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:06 smithi105 ceph-mon[27916]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T06:30:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:30:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:06 smithi177 ceph-mon[31346]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-22T06:30:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:30:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:07.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:07.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:07.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:09 smithi105 ceph-mon[27916]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:30:09.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:09.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:09 smithi112 ceph-mon[31236]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:09.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:30:09.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:09 smithi177 ceph-mon[31346]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T06:30:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:10 smithi105 ceph-mon[27916]: Deploying daemon osd.0 on smithi105 2024-04-22T06:30:10.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:10 smithi112 ceph-mon[31236]: Deploying daemon osd.0 on smithi105 2024-04-22T06:30:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:10 smithi177 ceph-mon[31346]: Deploying daemon osd.0 on smithi105 2024-04-22T06:30:11.761 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:11 smithi105 ceph-mon[27916]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:11 smithi112 ceph-mon[31236]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:11 smithi177 ceph-mon[31346]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:13 smithi105 ceph-mon[27916]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:13.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:13 smithi112 ceph-mon[31236]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:13.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:13 smithi177 ceph-mon[31346]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:15.756 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:15 smithi105 ceph-mon[27916]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:15.757 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.757 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.757 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:15 smithi112 ceph-mon[31236]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:15 smithi177 ceph-mon[31346]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:17.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:17.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:18.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.643+0000 7fb68bfff700 1 -- 172.21.15.105:0/1867761889 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb6a40ff110 con 0x7fb68c039890 2024-04-22T06:30:18.642 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 0 on host 'smithi105' 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 msgr2=0x7fb68c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 0x7fb68c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb698007be0 tx=0x7fb6980073d0 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 msgr2=0x7fb6a41058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb69400d980 tx=0x7fb69400dc90 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 shutdown_connections 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb68c039890 0x7fb68c03bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6a4103540 0x7fb6a41044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6a41019a0 0x7fb6a41058e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 --2- 172.21.15.105:0/1867761889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6a4102ba0 0x7fb6a4103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 >> 172.21.15.105:0/1867761889 conn(0x7fb6a40fcf40 msgr2=0x7fb6a40fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 shutdown_connections 2024-04-22T06:30:18.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:30:18.645+0000 7fb6a96d4700 1 -- 172.21.15.105:0/1867761889 wait complete. 2024-04-22T06:30:19.207 DEBUG:teuthology.orchestra.run.smithi105:osd.0> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T06:30:19.210 INFO:tasks.cephadm:Deploying osd.1 on smithi112 with /dev/vg_nvme/lv_4... 2024-04-22T06:30:19.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:30:19.457 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:30:19.893 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:19 smithi112 ceph-mon[31236]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:19.893 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.893 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.893 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:19.893 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:19 smithi112 ceph-mon[31236]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:30:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:19 smithi105 ceph-mon[27916]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:19.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:19 smithi105 ceph-mon[27916]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:30:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:19 smithi177 ceph-mon[31346]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:19.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:19 smithi177 ceph-mon[31346]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T06:30:20.782 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:20 smithi112 ceph-mon[31236]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:30:20.782 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:20 smithi112 ceph-mon[31236]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:30:20.783 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:20 smithi112 ceph-mon[31236]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:30:20.783 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:20.881 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:20 smithi105 ceph-mon[27916]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:30:20.882 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:20 smithi105 ceph-mon[27916]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:30:20.882 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:20 smithi105 ceph-mon[27916]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:30:20.882 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:20 smithi177 ceph-mon[31346]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T06:30:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:20 smithi177 ceph-mon[31346]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T06:30:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:20 smithi177 ceph-mon[31346]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:30:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116819 s, 898 MB/s 2024-04-22T06:30:21.351 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.746 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:21 smithi112 ceph-mon[31236]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' 2024-04-22T06:30:21.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:21.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:30:21.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:30:21.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-mon[27916]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' 2024-04-22T06:30:21.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0[42796]: 2024-04-22T06:30:21.566+0000 7fb7dadcc700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T06:30:21.919 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:30:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0[42796]: 2024-04-22T06:30:21.691+0000 7fb7d2bb7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:21 smithi177 ceph-mon[31346]: from='osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877]' entity='osd.0' 2024-04-22T06:30:22.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2024-04-22T06:30:22.291 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:22.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:22.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi105 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] boot 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:30:23.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:23.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- 172.21.15.112:0/1821882260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24103530 msgr2=0x7fba24105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 --2- 172.21.15.112:0/1821882260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24103530 0x7fba24105920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba18009990 tx=0x7fba18009ca0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- 172.21.15.112:0/1821882260 shutdown_connections 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 --2- 172.21.15.112:0/1821882260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24103530 0x7fba24105920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 --2- 172.21.15.112:0/1821882260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 0x7fba2410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 --2- 172.21.15.112:0/1821882260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24105e60 0x7fba24108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- 172.21.15.112:0/1821882260 >> 172.21.15.112:0/1821882260 conn(0x7fba240fcf60 msgr2=0x7fba240ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- 172.21.15.112:0/1821882260 shutdown_connections 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- 172.21.15.112:0/1821882260 wait complete. 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 Processor -- start 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.797+0000 7fba2a770700 1 -- start start 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24103530 0x7fba2419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:23.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 0x7fba241a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fba2410e5a0 con 0x7fba24103530 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fba2410e420 con 0x7fba24105e60 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba2a770700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fba2410e720 con 0x7fba24108790 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:35480/0 (socket says 172.21.15.112:35480) 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 -- 172.21.15.112:0/3773250654 learned_addr learned my addr 172.21.15.112:0/3773250654 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba23fff700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24103530 0x7fba2419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba28d0d700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 0x7fba241a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 -- 172.21.15.112:0/3773250654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 msgr2=0x7fba241a2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:23.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 0x7fba241a2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 -- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24103530 msgr2=0x7fba2419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24103530 0x7fba2419ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 -- 172.21.15.112:0/3773250654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba241a2ac0 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.798+0000 7fba237fe700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba1400d950 tx=0x7fba1400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.799+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14016070 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.799+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba1400fd40 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.799+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1400e3b0 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.799+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba241a2db0 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.799+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba241a3270 con 0x7fba24105e60 2024-04-22T06:30:23.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.800+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba24069730 con 0x7fba24105e60 2024-04-22T06:30:23.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.801+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fba14019070 con 0x7fba24105e60 2024-04-22T06:30:23.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.802+0000 7fba217fa700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 0x7fba0c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:23.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.802+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fba140ce310 con 0x7fba24105e60 2024-04-22T06:30:23.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.802+0000 7fba23fff700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 0x7fba0c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:23.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.803+0000 7fba23fff700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 0x7fba0c0a60f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b5c0 tx=0x7fba18005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:30:23.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:23.807+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba1409c760 con 0x7fba24105e60 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi105 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] boot 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:30:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:24.008 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:24.010+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fba24062e80 con 0x7fba0c0a3c30 2024-04-22T06:30:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi105 2024-04-22T06:30:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 17088M 2024-04-22T06:30:24.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T06:30:24.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:24.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: osd.0 [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] boot 2024-04-22T06:30:24.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T06:30:24.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:30:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:24 smithi112 ceph-mon[31236]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:30:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:24.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:24 smithi177 ceph-mon[31346]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:30:24.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:24.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:24.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:24 smithi105 ceph-mon[27916]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T06:30:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:25.859 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:25 smithi112 ceph-mon[31236]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:25.860 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:25 smithi112 ceph-mon[31236]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:25 smithi177 ceph-mon[31346]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:25 smithi177 ceph-mon[31346]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:25 smithi105 ceph-mon[27916]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:25 smithi105 ceph-mon[27916]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:26.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/376078851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:26.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:26 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:26.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:26 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]': finished 2024-04-22T06:30:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/376078851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:26 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:26 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]': finished 2024-04-22T06:30:27.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/376078851' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:27.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:26 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]: dispatch 2024-04-22T06:30:27.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:26 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4ded2dc-2b79-4cd1-a34a-82778a1120b0"}]': finished 2024-04-22T06:30:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:27 smithi177 ceph-mon[31346]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:27 smithi177 ceph-mon[31346]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:30:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:27 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/3880972668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:28.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:27 smithi105 ceph-mon[27916]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:28.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:27 smithi105 ceph-mon[27916]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:30:28.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:28.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:27 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/3880972668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:28.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:27 smithi112 ceph-mon[31236]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:28.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:27 smithi112 ceph-mon[31236]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T06:30:28.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:28.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:27 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/3880972668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:28 smithi177 ceph-mon[31346]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:28 smithi105 ceph-mon[27916]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:29.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:28 smithi112 ceph-mon[31236]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:31.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:31 smithi105 ceph-mon[27916]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:31.926 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:31 smithi112 ceph-mon[31236]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:31 smithi177 ceph-mon[31346]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:33.833 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:33 smithi112 ceph-mon[31236]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:33 smithi105 ceph-mon[27916]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:33.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:33 smithi177 ceph-mon[31346]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:34.901 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:34 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:30:34.902 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:34 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:34.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:34 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:30:34.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:34 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:34.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:34 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T06:30:34.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:34 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:35 smithi105 ceph-mon[27916]: Deploying daemon osd.1 on smithi112 2024-04-22T06:30:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:35 smithi105 ceph-mon[27916]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:35.936 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:35 smithi112 ceph-mon[31236]: Deploying daemon osd.1 on smithi112 2024-04-22T06:30:35.936 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:35 smithi112 ceph-mon[31236]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:35 smithi177 ceph-mon[31346]: Deploying daemon osd.1 on smithi112 2024-04-22T06:30:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:35 smithi177 ceph-mon[31346]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:37.906 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:37 smithi112 ceph-mon[31236]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:37.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:37 smithi105 ceph-mon[27916]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:37 smithi177 ceph-mon[31346]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:39.879 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:39 smithi112 ceph-mon[31236]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:39.880 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.880 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.880 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:39 smithi105 ceph-mon[27916]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:39 smithi177 ceph-mon[31346]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:41 smithi105 ceph-mon[27916]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:41.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:41 smithi112 ceph-mon[31236]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:41 smithi177 ceph-mon[31346]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:42.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.675+0000 7fba217fa700 1 -- 172.21.15.112:0/3773250654 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba24062e80 con 0x7fba0c0a3c30 2024-04-22T06:30:42.674 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 1 on host 'smithi112' 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.677+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 msgr2=0x7fba0c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.677+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 0x7fba0c0a60f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b5c0 tx=0x7fba18005fb0 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.677+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 msgr2=0x7fba2419b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.677+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba1400d950 tx=0x7fba1400dc60 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 shutdown_connections 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fba0c0a3c30 0x7fba0c0a60f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fba24108790 0x7fba241a2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fba24103530 0x7fba2419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 --2- 172.21.15.112:0/3773250654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba24105e60 0x7fba2419b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 >> 172.21.15.112:0/3773250654 conn(0x7fba240fcf60 msgr2=0x7fba24102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:30:42.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 shutdown_connections 2024-04-22T06:30:42.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:30:42.678+0000 7fba2a770700 1 -- 172.21.15.112:0/3773250654 wait complete. 2024-04-22T06:30:43.316 DEBUG:teuthology.orchestra.run.smithi112:osd.1> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T06:30:43.318 INFO:tasks.cephadm:Deploying osd.2 on smithi177 with /dev/vg_nvme/lv_4... 2024-04-22T06:30:43.318 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T06:30:43.459 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:30:43.382+0000 7f64ecc6e7c0 -1 osd.1 0 log_to_monitors true 2024-04-22T06:30:43.562 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:43.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:43 smithi112 ceph-mon[31236]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.899 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:43.900 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:43 smithi177 ceph-mon[31346]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:43.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:43.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:43 smithi105 ceph-mon[27916]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T06:30:44.849 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:44 smithi177 ceph-mon[31346]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:30:44.849 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:44 smithi177 ceph-mon[31346]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:30:44.849 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:44.849 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:44 smithi177 ceph-mon[31346]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:44.849 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:44 smithi177 ceph-mon[31346]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:44 smithi105 ceph-mon[27916]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:30:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:44 smithi105 ceph-mon[27916]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:30:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:44 smithi105 ceph-mon[27916]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:44 smithi105 ceph-mon[27916]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:45.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:30:44.722+0000 7f64e22d3700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T06:30:45.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:30:44.726+0000 7f64de0c6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:30:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-mon[31236]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T06:30:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-mon[31236]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T06:30:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-mon[31236]: from='osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:44 smithi112 ceph-mon[31236]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113471 s, 924 MB/s 2024-04-22T06:30:45.378 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.967 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:45.993 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-04-22T06:30:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:30:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T06:30:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:46.224 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:30:46.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:30:46.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] boot 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi112 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:46.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:46.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:46 smithi177 ceph-mon[31346]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:47.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:30:47.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] boot 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi112 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:47.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:46 smithi105 ceph-mon[27916]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:47.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:30:47.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:30:47.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: osd.1 [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] boot 2024-04-22T06:30:47.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi112 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 17088M 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:30:47.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:46 smithi112 ceph-mon[31236]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1721288592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 msgr2=0x7fd9ec10b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1721288592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec10b080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400b0a0 tx=0x7fd9e400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1721288592 shutdown_connections 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1721288592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec100db0 0x7fd9ec1011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1721288592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec10b080 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.787 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1721288592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec103e30 0x7fd9ec104290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1721288592 >> 172.21.15.177:0/1721288592 conn(0x7fd9ec0fc6f0 msgr2=0x7fd9ec0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:30:47.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1721288592 shutdown_connections 2024-04-22T06:30:47.788 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.788+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1721288592 wait complete. 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 Processor -- start 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 -- start start 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 0x7fd9ec103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec103e30 0x7fd9ec101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec10e570 con 0x7fd9ec100db0 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec10e3f0 con 0x7fd9ec103e30 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9f0d2f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec10e6f0 con 0x7fd9ec1047d0 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:57916/0 (socket says 172.21.15.177:57916) 2024-04-22T06:30:47.790 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 -- 172.21.15.177:0/1845886402 learned_addr learned my addr 172.21.15.177:0/1845886402 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9e3fff700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec103e30 0x7fd9ec101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ea59c700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 0x7fd9ec103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 -- 172.21.15.177:0/1845886402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec103e30 msgr2=0x7fd9ec101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec103e30 0x7fd9ec101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 -- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 msgr2=0x7fd9ec103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 0x7fd9ec103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 -- 172.21.15.177:0/1845886402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ec102bd0 con 0x7fd9ec1047d0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ea59c700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 0x7fd9ec103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.789+0000 7fd9ead9d700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400b070 tx=0x7fd9e4004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.790+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4021030 con 0x7fd9ec1047d0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.790+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ec1a2c80 con 0x7fd9ec1047d0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.790+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e4003a00 con 0x7fd9ec1047d0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.790+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4004600 con 0x7fd9ec1047d0 2024-04-22T06:30:47.791 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.790+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ec1a30e0 con 0x7fd9ec1047d0 2024-04-22T06:30:47.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.791+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ec069730 con 0x7fd9ec1047d0 2024-04-22T06:30:47.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.792+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7fd9e4024070 con 0x7fd9ec1047d0 2024-04-22T06:30:47.792 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.792+0000 7fd9e37fe700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 0x7fd9d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:30:47.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.793+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fd9e40d0230 con 0x7fd9ec1047d0 2024-04-22T06:30:47.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.793+0000 7fd9ea59c700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 0x7fd9d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:30:47.793 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.793+0000 7fd9ea59c700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 0x7fd9d80a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc005950 tx=0x7fd9dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:30:47.795 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:47.796+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9e409e4f0 con 0x7fd9ec1047d0 2024-04-22T06:30:48.000 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:30:48.001+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd9ec062e80 con 0x7fd9d80a3bb0 2024-04-22T06:30:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:48 smithi105 ceph-mon[27916]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:30:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:48.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:48.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:48 smithi112 ceph-mon[31236]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:30:48.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:48.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:48.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:48.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:48 smithi177 ceph-mon[31346]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T06:30:48.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:30:48.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:30:48.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:49 smithi105 ceph-mon[27916]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:49 smithi105 ceph-mon[27916]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:49.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:49 smithi177 ceph-mon[31346]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:49.695 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:49 smithi177 ceph-mon[31346]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:49.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:49 smithi112 ceph-mon[31236]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:30:49.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:49 smithi112 ceph-mon[31236]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/1780998721' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:50 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:50 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]': finished 2024-04-22T06:30:50.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:50 smithi177 ceph-mon[31346]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:30:50.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:30:50.787 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/1780998721' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.787 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:50 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.787 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:50 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]': finished 2024-04-22T06:30:50.787 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:50 smithi105 ceph-mon[27916]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:30:50.787 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:30:50.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/1780998721' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:50 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]: dispatch 2024-04-22T06:30:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:50 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d48dd22-4dc5-4b82-bb01-119c1777318c"}]': finished 2024-04-22T06:30:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:50 smithi112 ceph-mon[31236]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T06:30:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:30:51.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:51 smithi177 ceph-mon[31346]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:51.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:51 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/924623974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:51 smithi105 ceph-mon[27916]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/924623974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:51.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:51 smithi112 ceph-mon[31236]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:51 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/924623974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:30:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:53 smithi105 ceph-mon[27916]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:53 smithi112 ceph-mon[31236]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:53 smithi177 ceph-mon[31346]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:55.842 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:55 smithi177 ceph-mon[31346]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:55 smithi105 ceph-mon[27916]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:55 smithi112 ceph-mon[31236]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:57.894 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:57 smithi177 ceph-mon[31346]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:57 smithi105 ceph-mon[27916]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:57.948 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:57 smithi112 ceph-mon[31236]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:58.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:58 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:30:58.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:58 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:58.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:58 smithi105 ceph-mon[27916]: Deploying daemon osd.2 on smithi177 2024-04-22T06:30:58.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:58 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:30:58.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:58 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:58.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:58 smithi112 ceph-mon[31236]: Deploying daemon osd.2 on smithi177 2024-04-22T06:30:58.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:58 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T06:30:58.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:58 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:30:58.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:58 smithi177 ceph-mon[31346]: Deploying daemon osd.2 on smithi177 2024-04-22T06:30:59.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:30:59 smithi177 ceph-mon[31346]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:59.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:30:59 smithi105 ceph-mon[27916]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:30:59.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:30:59 smithi112 ceph-mon[31236]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:01.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:01 smithi105 ceph-mon[27916]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:01 smithi112 ceph-mon[31236]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:01 smithi177 ceph-mon[31346]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:03.693 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:03 smithi177 ceph-mon[31346]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:03.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:03.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:03.694 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:03 smithi105 ceph-mon[27916]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:03.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:03.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:03 smithi112 ceph-mon[31236]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:03.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:03.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:03.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:05.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:05 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:05.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:05.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:05 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:05 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:06.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.059+0000 7fd9e37fe700 1 -- 172.21.15.177:0/1845886402 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd9ec062e80 con 0x7fd9d80a3bb0 2024-04-22T06:31:06.058 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 2 on host 'smithi177' 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 msgr2=0x7fd9d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 0x7fd9d80a6070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc005950 tx=0x7fd9dc0058e0 comp rx=0 tx=0).stop 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 msgr2=0x7fd9ec102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400b070 tx=0x7fd9e4004e40 comp rx=0 tx=0).stop 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 shutdown_connections 2024-04-22T06:31:06.060 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9d80a3bb0 0x7fd9d80a6070 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9ec1047d0 0x7fd9ec102320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9ec100db0 0x7fd9ec103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 --2- 172.21.15.177:0/1845886402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec103e30 0x7fd9ec101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 >> 172.21.15.177:0/1845886402 conn(0x7fd9ec0fc6f0 msgr2=0x7fd9ec107130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 shutdown_connections 2024-04-22T06:31:06.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:31:06.062+0000 7fd9f0d2f700 1 -- 172.21.15.177:0/1845886402 wait complete. 2024-04-22T06:31:06.652 DEBUG:teuthology.orchestra.run.smithi177:osd.2> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T06:31:06.654 INFO:tasks.cephadm:Deploying osd.3 on smithi105 with /dev/vg_nvme/lv_3... 2024-04-22T06:31:06.654 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:31:06.901 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:07.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:07 smithi177 ceph-mon[31346]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:07.340 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:07 smithi105 ceph-mon[27916]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:07.457 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:07 smithi112 ceph-mon[31236]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T06:31:08.127 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:08 smithi105 ceph-mon[27916]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:31:08.127 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:08 smithi105 ceph-mon[27916]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:31:08.127 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:08 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:08.127 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:08 smithi105 ceph-mon[27916]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:08.127 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:08 smithi105 ceph-mon[27916]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:08.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-mon[31346]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:31:08.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-mon[31346]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:31:08.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:08.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-mon[31346]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:08.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-mon[31346]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:08.241 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2[36481]: 2024-04-22T06:31:08.058+0000 7f0426861700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T06:31:08.241 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:08 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2[36481]: 2024-04-22T06:31:08.063+0000 7f0422654700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:31:08.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:08 smithi112 ceph-mon[31236]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T06:31:08.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:08 smithi112 ceph-mon[31236]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T06:31:08.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:08 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:08.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:08 smithi112 ceph-mon[31236]: from='osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:08.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:08 smithi112 ceph-mon[31236]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:31:09.002 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:31:09.002 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:31:09.003 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-22T06:31:09.003 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-22T06:31:09.003 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111171 s, 943 MB/s 2024-04-22T06:31:09.003 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-22T06:31:09.283 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:09 smithi105 ceph-mon[27916]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:31:09.283 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:09 smithi105 ceph-mon[27916]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:31:09.283 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.283 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.283 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:09 smithi105 ceph-mon[27916]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:09.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:09 smithi177 ceph-mon[31346]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:31:09.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:09 smithi177 ceph-mon[31346]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:31:09.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.305 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:09 smithi177 ceph-mon[31346]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:09.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:09 smithi112 ceph-mon[31236]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:31:09.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:09 smithi112 ceph-mon[31236]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T06:31:09.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:09.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:09 smithi112 ceph-mon[31236]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-22T06:31:09.621 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-04-22T06:31:09.860 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] boot 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi177 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.234 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:10.235 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:10.235 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] boot 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi177 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:10.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:10.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: osd.2 [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] boot 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi177 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 17088M 2024-04-22T06:31:10.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:10.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:10.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:10.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 -- 172.21.15.105:0/994776658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 msgr2=0x7f271010af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 --2- 172.21.15.105:0/994776658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f271010af20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f270800b0d0 tx=0x7f270800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 -- 172.21.15.105:0/994776658 shutdown_connections 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 --2- 172.21.15.105:0/994776658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710102ad0 0x7f2710102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 --2- 172.21.15.105:0/994776658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f271010af20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 --2- 172.21.15.105:0/994776658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710103cd0 0x7f2710104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 -- 172.21.15.105:0/994776658 >> 172.21.15.105:0/994776658 conn(0x7f27100fe090 msgr2=0x7f27101004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 -- 172.21.15.105:0/994776658 shutdown_connections 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.484+0000 7f2717317700 1 -- 172.21.15.105:0/994776658 wait complete. 2024-04-22T06:31:11.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 Processor -- start 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 -- start start 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710102ad0 0x7f2710198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710103cd0 0x7f2710199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f271010e600 con 0x7f2710104670 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f271010e480 con 0x7f2710103cd0 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f2717317700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f271010e780 con 0x7f2710102ad0 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39908/0 (socket says 172.21.15.105:39908) 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 -- 172.21.15.105:0/3711210038 learned_addr learned my addr 172.21.15.105:0/3711210038 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 -- 172.21.15.105:0/3711210038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710102ad0 msgr2=0x7f2710198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27150b3700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710102ad0 0x7f2710198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710102ad0 0x7f2710198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 -- 172.21.15.105:0/3711210038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710103cd0 msgr2=0x7f2710199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:11.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27148b2700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710103cd0 0x7f2710199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710103cd0 0x7f2710199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 -- 172.21.15.105:0/3711210038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27101a08a0 con 0x7f2710104670 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.485+0000 7f27158b4700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2708009fd0 tx=0x7f2708004e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.486+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270801f030 con 0x7f2710104670 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.486+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27101a0b30 con 0x7f2710104670 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.486+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2708007a80 con 0x7f2710104670 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.486+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708004810 con 0x7f2710104670 2024-04-22T06:31:11.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.486+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27101a0f80 con 0x7f2710104670 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.487+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271002df80 con 0x7f2710104670 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.488+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f2708003a00 con 0x7f2710104670 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.489+0000 7f27067fc700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.489+0000 7f27150b3700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.489+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f270800f020 con 0x7f2710104670 2024-04-22T06:31:11.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.489+0000 7f27150b3700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f271019a130 tx=0x7f2700008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:31:11.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.493+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2708029020 con 0x7f2710104670 2024-04-22T06:31:11.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:11 smithi105 ceph-mon[27916]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:31:11.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:11 smithi105 ceph-mon[27916]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:11.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:11.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:11.703+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2710062e80 con 0x7f26fc0a3bb0 2024-04-22T06:31:11.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:11 smithi112 ceph-mon[31236]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:31:11.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:11 smithi112 ceph-mon[31236]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:11.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:11 smithi177 ceph-mon[31346]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T06:31:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:11 smithi177 ceph-mon[31346]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47331]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47331]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47331]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47331]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47334]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47334]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47334]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.623 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47334]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:12.624 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:12.637 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:12.637 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:31:12.637 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:12.638 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:12.638 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:12.638 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:12.638 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40088]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40088]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40088]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40088]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40091]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40091]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.638 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40091]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:12.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39851]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:12.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39851]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39851]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39854]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39854]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39854]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47350]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47350]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47350]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47350]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47353]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47353]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47353]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:12 smithi105 sudo[47353]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.990 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40091]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40094]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40094]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40094]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40097]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:12.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40097]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:12.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40097]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:12.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:12 smithi177 sudo[40097]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:13.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39857]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39857]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39857]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39857]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39860]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39860]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-22T06:31:13.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:12 smithi112 sudo[39860]: pam_unix(sudo:session): session closed for user root 2024-04-22T06:31:13.502 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:13.502 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:31:13.502 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.503 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.504 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.504 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:13 smithi105 ceph-mon[27916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:13.546+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f270809a530 con 0x7f2710104670 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:13 smithi112 ceph-mon[31236]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:31:13.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:13 smithi177 ceph-mon[31346]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T06:31:14.656 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T06:31:14.656 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:31:14.657 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]: dispatch 2024-04-22T06:31:14.657 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]': finished 2024-04-22T06:31:14.657 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:31:14.657 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:14.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T06:31:14.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:31:14.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]: dispatch 2024-04-22T06:31:14.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]': finished 2024-04-22T06:31:14.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:31:14.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]: dispatch 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/513890155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b765227c-f85b-48e2-bc46-0de059180664"}]': finished 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T06:31:14.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:15.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:15 smithi105 ceph-mon[27916]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:15.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:15 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/123848747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:15.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:15 smithi112 ceph-mon[31236]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:15.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:15 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/123848747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:15.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:15 smithi177 ceph-mon[31346]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:15.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:15 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/123848747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:17.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:17 smithi105 ceph-mon[27916]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:17.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:17 smithi112 ceph-mon[31236]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:17 smithi177 ceph-mon[31346]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:19.830 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:19 smithi105 ceph-mon[27916]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:19 smithi112 ceph-mon[31236]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:19 smithi177 ceph-mon[31346]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:21.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:21 smithi105 ceph-mon[27916]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:21 smithi112 ceph-mon[31236]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:21 smithi177 ceph-mon[31346]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:22.811 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:31:22.812 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:31:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T06:31:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:23.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:23 smithi105 ceph-mon[27916]: Deploying daemon osd.3 on smithi105 2024-04-22T06:31:23.856 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:23 smithi105 ceph-mon[27916]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:23.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:23 smithi112 ceph-mon[31236]: Deploying daemon osd.3 on smithi105 2024-04-22T06:31:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:23 smithi112 ceph-mon[31236]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:23 smithi177 ceph-mon[31346]: Deploying daemon osd.3 on smithi105 2024-04-22T06:31:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:23 smithi177 ceph-mon[31346]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:25.813 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:25 smithi105 ceph-mon[27916]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:25 smithi112 ceph-mon[31236]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:25 smithi177 ceph-mon[31346]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:27.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:27 smithi105 ceph-mon[27916]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:27.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:27.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:27 smithi112 ceph-mon[31236]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:27.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:27 smithi177 ceph-mon[31346]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:29.838 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:29 smithi105 ceph-mon[27916]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:29.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:29 smithi112 ceph-mon[31236]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:29 smithi177 ceph-mon[31346]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:31.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.170+0000 7f27067fc700 1 -- 172.21.15.105:0/3711210038 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2710062e80 con 0x7f26fc0a3bb0 2024-04-22T06:31:31.169 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 3 on host 'smithi105' 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 msgr2=0x7f26fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f271019a130 tx=0x7f2700008040 comp rx=0 tx=0).stop 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 msgr2=0x7f27101a01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2708009fd0 tx=0x7f2708004e00 comp rx=0 tx=0).stop 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 shutdown_connections 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:31.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2710104670 0x7f27101a01d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:31.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2710102ad0 0x7f2710198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:31.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 --2- 172.21.15.105:0/3711210038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2710103cd0 0x7f2710199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:31.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 >> 172.21.15.105:0/3711210038 conn(0x7f27100fe090 msgr2=0x7f2710106f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:31:31.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 shutdown_connections 2024-04-22T06:31:31.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:31:31.172+0000 7f2717317700 1 -- 172.21.15.105:0/3711210038 wait complete. 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.415 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.416 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:31 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:31 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:31.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:31.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:31 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:31.825 DEBUG:teuthology.orchestra.run.smithi105:osd.3> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T06:31:31.827 INFO:tasks.cephadm:Deploying osd.4 on smithi112 with /dev/vg_nvme/lv_3... 2024-04-22T06:31:31.827 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:31:32.071 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:31:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:32 smithi105 ceph-mon[27916]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:31:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:32 smithi112 ceph-mon[31236]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:31:32.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:32 smithi177 ceph-mon[31346]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T06:31:33.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:33 smithi112 ceph-mon[31236]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:31:33.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:33 smithi112 ceph-mon[31236]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:31:33.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:33 smithi112 ceph-mon[31236]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:31:33.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:33.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:33 smithi112 ceph-mon[31236]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:33.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-mon[27916]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:31:33.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-mon[27916]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:31:33.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-mon[27916]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:31:33.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:33.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-mon[27916]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:33.669 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:31:33.271+0000 7ff1d33cf700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T06:31:33.669 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:31:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:31:33.280+0000 7ff1cf1c2700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:31:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:33 smithi177 ceph-mon[31346]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T06:31:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:33 smithi177 ceph-mon[31346]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T06:31:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:33 smithi177 ceph-mon[31346]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:31:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:33 smithi177 ceph-mon[31346]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-22T06:31:33.851 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:31:33.851 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:31:33.851 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-22T06:31:33.852 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-22T06:31:33.852 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114192 s, 918 MB/s 2024-04-22T06:31:33.852 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] boot 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:31:34.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:34 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.524 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2024-04-22T06:31:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:31:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T06:31:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] boot 2024-04-22T06:31:34.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:31:34.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:34 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: from='osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: osd.3 [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] boot 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T06:31:34.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:34 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:31:34.763 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi105 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:35 smithi112 ceph-mon[31236]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:35.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi105 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:35 smithi105 ceph-mon[27916]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi105 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 8544M 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:35 smithi177 ceph-mon[31346]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 -- 172.21.15.112:0/1144548398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 msgr2=0x7fa56410aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 --2- 172.21.15.112:0/1144548398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa56410aef0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00b0d0 tx=0x7fa55c00b3e0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 -- 172.21.15.112:0/1144548398 shutdown_connections 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 --2- 172.21.15.112:0/1144548398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa56410aef0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 --2- 172.21.15.112:0/1144548398 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 0x7fa564102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 --2- 172.21.15.112:0/1144548398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 0x7fa564104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.457 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 -- 172.21.15.112:0/1144548398 >> 172.21.15.112:0/1144548398 conn(0x7fa5640fe040 msgr2=0x7fa564100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:31:36.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.457+0000 7fa56ad46700 1 -- 172.21.15.112:0/1144548398 shutdown_connections 2024-04-22T06:31:36.458 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 -- 172.21.15.112:0/1144548398 wait complete. 2024-04-22T06:31:36.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:36 smithi112 ceph-mon[31236]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 Processor -- start 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 -- start start 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 0x7fa5640767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 0x7fa564074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa56410e520 con 0x7fa564103ca0 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa56410e3a0 con 0x7fa564104640 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa56ad46700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa56410e6a0 con 0x7fa564102aa0 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa5692e3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa5692e3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34486/0 (socket says 172.21.15.112:34486) 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.458+0000 7fa5692e3700 1 -- 172.21.15.112:0/3778092836 learned_addr learned my addr 172.21.15.112:0/3778092836 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-22T06:31:36.460 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa568ae2700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 0x7fa5640767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa563fff700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 0x7fa564074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 -- 172.21.15.112:0/3778092836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 msgr2=0x7fa5640767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 0x7fa5640767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 -- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 msgr2=0x7fa564074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 0x7fa564074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 -- 172.21.15.112:0/3778092836 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa564075c20 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa5692e3700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009fd0 tx=0x7fa55c004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c01f030 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa55c003a00 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5641a2c10 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0047f0 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.459+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5641a3060 con 0x7fa564104640 2024-04-22T06:31:36.461 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.460+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56402df80 con 0x7fa564104640 2024-04-22T06:31:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.461+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa55c003bb0 con 0x7fa564104640 2024-04-22T06:31:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.462+0000 7fa561ffb700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 0x7fa5540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:31:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.462+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7fa55c00f020 con 0x7fa564104640 2024-04-22T06:31:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.462+0000 7fa568ae2700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 0x7fa5540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:31:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.463+0000 7fa568ae2700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 0x7fa5540a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa564103b00 tx=0x7fa55800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:31:36.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.467+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa55c029020 con 0x7fa564104640 2024-04-22T06:31:36.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:36 smithi105 ceph-mon[27916]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:31:36.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:36.673+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa564062e80 con 0x7fa5540a3cc0 2024-04-22T06:31:36.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:36 smithi177 ceph-mon[31346]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T06:31:37.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:31:37.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:37.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:37.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:37.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:37.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:37.639 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:31:37.640 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:37.640 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:37.640 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:37.640 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:37.640 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:37.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: osdmap e28: 4 total, 4 up, 4 in 2024-04-22T06:31:37.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:37.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:31:37.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:31:37.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:31:37.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:38.528 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:38 smithi112 ceph-mon[31236]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:31:38.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:38 smithi105 ceph-mon[27916]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:31:38.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:38 smithi177 ceph-mon[31346]: osdmap e29: 4 total, 4 up, 4 in 2024-04-22T06:31:39.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:39 smithi105 ceph-mon[27916]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T06:31:39.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:39 smithi105 ceph-mon[27916]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:31:39.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:39 smithi112 ceph-mon[31236]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T06:31:39.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:39 smithi112 ceph-mon[31236]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:31:39.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:39 smithi177 ceph-mon[31346]: osdmap e30: 4 total, 4 up, 4 in 2024-04-22T06:31:39.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:39 smithi177 ceph-mon[31346]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T06:31:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/16532545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]': finished 2024-04-22T06:31:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:31:40.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:40.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/1647837316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/16532545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]': finished 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/1647837316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/16532545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]: dispatch 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ae3793c-2262-486a-8754-ab46bcea415d"}]': finished 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: osdmap e31: 5 total, 4 up, 5 in 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/1647837316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:31:41.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:41 smithi105 ceph-mon[27916]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-22T06:31:41.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:41 smithi112 ceph-mon[31236]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-22T06:31:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:41 smithi177 ceph-mon[31346]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-22T06:31:43.858 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:43 smithi112 ceph-mon[31236]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:31:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:43 smithi105 ceph-mon[27916]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:31:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:43 smithi177 ceph-mon[31346]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:31:45.816 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:45 smithi112 ceph-mon[31236]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:45.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:45 smithi105 ceph-mon[27916]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:45.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:45 smithi177 ceph-mon[31346]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:47.832 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:47 smithi112 ceph-mon[31236]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:47.832 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:31:47.832 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:47 smithi105 ceph-mon[27916]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:31:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:47 smithi177 ceph-mon[31346]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T06:31:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:48.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:48 smithi112 ceph-mon[31236]: Deploying daemon osd.4 on smithi112 2024-04-22T06:31:48.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:48 smithi105 ceph-mon[27916]: Deploying daemon osd.4 on smithi112 2024-04-22T06:31:48.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:48 smithi177 ceph-mon[31346]: Deploying daemon osd.4 on smithi112 2024-04-22T06:31:49.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:49 smithi105 ceph-mon[27916]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:49.923 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:49 smithi112 ceph-mon[31236]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:49 smithi177 ceph-mon[31346]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:51.773 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:51 smithi112 ceph-mon[31236]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:51.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:51 smithi105 ceph-mon[27916]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:51 smithi177 ceph-mon[31346]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:53.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:53 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:53 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:53 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:53.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:53 smithi112 ceph-mon[31236]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:53.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:53 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:53 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:53 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:53.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:53 smithi105 ceph-mon[27916]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:53.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:53 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:53 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:53.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:53 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:53.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:53 smithi177 ceph-mon[31346]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:55.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:55.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:55.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:31:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:56.037 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 4 on host 'smithi112' 2024-04-22T06:31:56.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.037+0000 7fa561ffb700 1 -- 172.21.15.112:0/3778092836 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa564062e80 con 0x7fa5540a3cc0 2024-04-22T06:31:56.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 msgr2=0x7fa5540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:56.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 0x7fa5540a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa564103b00 tx=0x7fa55800a380 comp rx=0 tx=0).stop 2024-04-22T06:31:56.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 msgr2=0x7fa564075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:31:56.039 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009fd0 tx=0x7fa55c004e40 comp rx=0 tx=0).stop 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 shutdown_connections 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5540a3cc0 0x7fa5540a6180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa564104640 0x7fa564075370 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa564102aa0 0x7fa5640767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 --2- 172.21.15.112:0/3778092836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa564103ca0 0x7fa564074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 >> 172.21.15.112:0/3778092836 conn(0x7fa5640fe040 msgr2=0x7fa564106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 shutdown_connections 2024-04-22T06:31:56.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:31:56.040+0000 7fa56ad46700 1 -- 172.21.15.112:0/3778092836 wait complete. 2024-04-22T06:31:56.688 DEBUG:teuthology.orchestra.run.smithi112:osd.4> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T06:31:56.690 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_3... 2024-04-22T06:31:56.690 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T06:31:56.937 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:31:57.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.353 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:57 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:57.354 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:57 smithi177 ceph-mon[31346]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:31:57.354 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:57 smithi177 ceph-mon[31346]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:57.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:57 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:57.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:57 smithi105 ceph-mon[27916]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:31:57.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:57 smithi105 ceph-mon[27916]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:57.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:31:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:57 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:31:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:57 smithi112 ceph-mon[31236]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T06:31:57.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:57 smithi112 ceph-mon[31236]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:58.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-mon[31236]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:31:58.143 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-mon[31236]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:31:58.143 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-mon[31236]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:31:58.143 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:58.143 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:31:58.046+0000 7f9df8124700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T06:31:58.143 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:31:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:31:58.052+0000 7f9df3f17700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:31:58.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:58 smithi177 ceph-mon[31346]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:31:58.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:58 smithi177 ceph-mon[31346]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:31:58.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:58 smithi177 ceph-mon[31346]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:31:58.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:58 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:58.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:58 smithi105 ceph-mon[27916]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T06:31:58.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:58 smithi105 ceph-mon[27916]: osdmap e32: 5 total, 4 up, 5 in 2024-04-22T06:31:58.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:58 smithi105 ceph-mon[27916]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:31:58.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:58 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111765 s, 938 MB/s 2024-04-22T06:31:58.713 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-22T06:31:59.205 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-04-22T06:31:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:59 smithi112 ceph-mon[31236]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:31:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:59 smithi112 ceph-mon[31236]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T06:31:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:31:59 smithi112 ceph-mon[31236]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:59.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:59 smithi105 ceph-mon[27916]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:31:59.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:59 smithi105 ceph-mon[27916]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T06:31:59.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:31:59 smithi105 ceph-mon[27916]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:31:59.439 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:31:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:59 smithi177 ceph-mon[31346]: from='osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:31:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:59 smithi177 ceph-mon[31346]: osdmap e33: 5 total, 4 up, 5 in 2024-04-22T06:31:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:31:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:31:59 smithi177 ceph-mon[31346]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] boot 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi112 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.235 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:00.236 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] boot 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi112 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:00.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:00.420 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: osd.4 [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] boot 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi112 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 8544M 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:00.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 -- 172.21.15.177:0/1278394873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4103cd0 msgr2=0x7f46f4104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 --2- 172.21.15.177:0/1278394873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4103cd0 0x7f46f4104130 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46e8009990 tx=0x7f46e8009ca0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 -- 172.21.15.177:0/1278394873 shutdown_connections 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 --2- 172.21.15.177:0/1278394873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4104670 0x7f46f410af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 --2- 172.21.15.177:0/1278394873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 0x7f46f4102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 --2- 172.21.15.177:0/1278394873 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4103cd0 0x7f46f4104130 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 -- 172.21.15.177:0/1278394873 >> 172.21.15.177:0/1278394873 conn(0x7f46f40fe090 msgr2=0x7f46f41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 -- 172.21.15.177:0/1278394873 shutdown_connections 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.034+0000 7f46f98ad700 1 -- 172.21.15.177:0/1278394873 wait complete. 2024-04-22T06:32:01.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 Processor -- start 2024-04-22T06:32:01.035 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 -- start start 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 0x7f46f40767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4103cd0 0x7f46f4074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e5c0 con 0x7f46f4102ad0 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e440 con 0x7f46f4103cd0 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.035+0000 7f46f98ad700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46f410e740 con 0x7f46f4104670 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46686/0 (socket says 172.21.15.177:46686) 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 -- 172.21.15.177:0/3423056820 learned_addr learned my addr 172.21.15.177:0/3423056820 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:32:01.037 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f2ffd700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 0x7f46f40767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f27fc700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4103cd0 0x7f46f4074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 -- 172.21.15.177:0/3423056820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4103cd0 msgr2=0x7f46f4074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4103cd0 0x7f46f4074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 -- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 msgr2=0x7f46f40767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 0x7f46f40767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 -- 172.21.15.177:0/3423056820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f4075c20 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f37fe700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00b970 tx=0x7f46ec00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec019070 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f41a2d00 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec00cde0 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec007370 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.036+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f41a31e0 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.037+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f4069730 con 0x7f46f4104670 2024-04-22T06:32:01.038 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.038+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f46ec01c070 con 0x7f46f4104670 2024-04-22T06:32:01.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.039+0000 7f46f88ab700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:01.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.039+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f46ec0ce810 con 0x7f46f4104670 2024-04-22T06:32:01.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.039+0000 7f46f2ffd700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:01.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.039+0000 7f46f2ffd700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 0x7f46dc0a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46f4103b30 tx=0x7f46e400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:01.042 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.044+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46ec09c260 con 0x7f46f4104670 2024-04-22T06:32:01.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:01 smithi177 ceph-mon[31346]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T06:32:01.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:01 smithi177 ceph-mon[31346]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:01.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:01.252+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f46f402e3d0 con 0x7f46dc0a3c40 2024-04-22T06:32:01.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:01 smithi105 ceph-mon[27916]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T06:32:01.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:01 smithi105 ceph-mon[27916]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:01.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:01 smithi112 ceph-mon[31236]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T06:32:01.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:01 smithi112 ceph-mon[31236]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:02.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:02 smithi177 ceph-mon[31346]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:02.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:02 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:02.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:02 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:02.124 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:02 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:02 smithi105 ceph-mon[27916]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:02 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:02 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:02 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:02.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:02 smithi112 ceph-mon[31236]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:02.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:02 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:02.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:02 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:02.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:02 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:03.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:03 smithi105 ceph-mon[27916]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:03.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:03 smithi112 ceph-mon[31236]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:03 smithi177 ceph-mon[31346]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:04.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/557405752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:04 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:04 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]': finished 2024-04-22T06:32:04.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:04 smithi105 ceph-mon[27916]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:32:04.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:04 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/557405752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:04 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:04 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]': finished 2024-04-22T06:32:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:04 smithi112 ceph-mon[31236]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:32:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:04 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/557405752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:04 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]: dispatch 2024-04-22T06:32:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:04 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9fd1a3-a0ef-4d21-afed-1aca4e529af0"}]': finished 2024-04-22T06:32:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:04 smithi177 ceph-mon[31346]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T06:32:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:04 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:05.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:05 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/518420207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:05.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:05 smithi105 ceph-mon[27916]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:05.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:05 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/518420207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:05.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:05 smithi112 ceph-mon[31236]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:05.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:05 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/518420207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:05.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:05 smithi177 ceph-mon[31346]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:07.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:07 smithi105 ceph-mon[27916]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:07.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:07 smithi112 ceph-mon[31236]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:07.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:07 smithi177 ceph-mon[31346]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:09.757 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:09 smithi177 ceph-mon[31346]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:09 smithi105 ceph-mon[27916]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:09.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:09 smithi112 ceph-mon[31236]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:11.916 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:11 smithi177 ceph-mon[31346]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:11.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:32:11.917 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:11 smithi105 ceph-mon[27916]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:32:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:11 smithi112 ceph-mon[31236]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T06:32:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:12.834 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:12 smithi177 ceph-mon[31346]: Deploying daemon osd.5 on smithi177 2024-04-22T06:32:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:12 smithi105 ceph-mon[27916]: Deploying daemon osd.5 on smithi177 2024-04-22T06:32:12.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:12 smithi112 ceph-mon[31236]: Deploying daemon osd.5 on smithi177 2024-04-22T06:32:13.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:13 smithi177 ceph-mon[31346]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:13.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:13 smithi105 ceph-mon[27916]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:13.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:13 smithi112 ceph-mon[31236]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:15.783 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:15 smithi177 ceph-mon[31346]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:15.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:15 smithi105 ceph-mon[27916]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:15.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:15 smithi112 ceph-mon[31236]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:16.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:16.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:16.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:16.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:17.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:17 smithi105 ceph-mon[27916]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:17.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:17 smithi112 ceph-mon[31236]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:17 smithi177 ceph-mon[31346]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:19.617 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-04-22T06:32:19.617 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.618+0000 7f46f88ab700 1 -- 172.21.15.177:0/3423056820 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f46f402e3d0 con 0x7f46dc0a3c40 2024-04-22T06:32:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 msgr2=0x7f46dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 0x7f46dc0a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46f4103b30 tx=0x7f46e400a3b0 comp rx=0 tx=0).stop 2024-04-22T06:32:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 msgr2=0x7f46f4075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00b970 tx=0x7f46ec00bc80 comp rx=0 tx=0).stop 2024-04-22T06:32:19.619 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 shutdown_connections 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46dc0a3c40 0x7f46dc0a6100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46f4104670 0x7f46f4075370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46f4102ad0 0x7f46f40767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 --2- 172.21.15.177:0/3423056820 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f4103cd0 0x7f46f4074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 >> 172.21.15.177:0/3423056820 conn(0x7f46f40fe090 msgr2=0x7f46f4106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 shutdown_connections 2024-04-22T06:32:19.620 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:32:19.620+0000 7f46f98ad700 1 -- 172.21.15.177:0/3423056820 wait complete. 2024-04-22T06:32:19.871 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:19 smithi177 ceph-mon[31346]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:19.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:19.872 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:19 smithi105 ceph-mon[27916]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:19 smithi112 ceph-mon[31236]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.191 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T06:32:20.193 INFO:tasks.cephadm:Deploying osd.6 on smithi105 with /dev/vg_nvme/lv_2... 2024-04-22T06:32:20.193 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:32:20.465 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:32:20.490 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:32:20 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:32:20.273+0000 7f28303fc7c0 -1 osd.5 0 log_to_monitors true 2024-04-22T06:32:20.777 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:20 smithi105 ceph-mon[27916]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.778 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:20 smithi105 ceph-mon[27916]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.778 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.778 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.945 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:20 smithi177 ceph-mon[31346]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:20 smithi177 ceph-mon[31346]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:20 smithi112 ceph-mon[31236]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:20 smithi112 ceph-mon[31236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T06:32:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:20.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:21.817 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:21 smithi105 ceph-mon[27916]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:21.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:21.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:32:21.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:21.955 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:32:21.956 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:21.956 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:21 smithi177 ceph-mon[31346]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: from='osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:21.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:21 smithi112 ceph-mon[31236]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0186144 s, 563 MB/s 2024-04-22T06:32:22.520 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-22T06:32:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:22 smithi105 ceph-mon[27916]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:32:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:22 smithi105 ceph-mon[27916]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T06:32:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:22 smithi105 ceph-mon[27916]: from='osd.5 ' entity='osd.5' 2024-04-22T06:32:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:22 smithi112 ceph-mon[31236]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:32:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:22 smithi112 ceph-mon[31236]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T06:32:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:22 smithi112 ceph-mon[31236]: from='osd.5 ' entity='osd.5' 2024-04-22T06:32:22.990 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:32:22.594+0000 7f2825a61700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T06:32:22.990 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:32:22.764+0000 7f281d84c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:32:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-mon[31346]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:32:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-mon[31346]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T06:32:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:22.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:22 smithi177 ceph-mon[31346]: from='osd.5 ' entity='osd.5' 2024-04-22T06:32:23.234 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-04-22T06:32:23.468 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] boot 2024-04-22T06:32:23.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:32:23.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.910 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] boot 2024-04-22T06:32:23.910 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:32:23.910 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: osdmap e39: 6 total, 5 up, 6 in 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: osd.5 [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] boot 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T06:32:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 -- 172.21.15.105:0/3100981516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314108790 msgr2=0x7f031410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 --2- 172.21.15.105:0/3100981516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314108790 0x7f031410ac00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f030c00b0a0 tx=0x7f030c00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 -- 172.21.15.105:0/3100981516 shutdown_connections 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 --2- 172.21.15.105:0/3100981516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314103530 0x7f0314105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 --2- 172.21.15.105:0/3100981516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314108790 0x7f031410ac00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 --2- 172.21.15.105:0/3100981516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314105e60 0x7f0314108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 -- 172.21.15.105:0/3100981516 >> 172.21.15.105:0/3100981516 conn(0x7f03140fcf60 msgr2=0x7f03140ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:32:25.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 -- 172.21.15.105:0/3100981516 shutdown_connections 2024-04-22T06:32:25.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 -- 172.21.15.105:0/3100981516 wait complete. 2024-04-22T06:32:25.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.089+0000 7f0319431700 1 Processor -- start 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 -- start start 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 0x7f03141014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314108790 0x7f0314101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f031410e530 con 0x7f0314105e60 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f031410e3b0 con 0x7f0314108790 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0319431700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f031410e6b0 con 0x7f0314103530 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0312ffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0312ffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:58564/0 (socket says 172.21.15.105:58564) 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 0x7f03141014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:25.091 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0312ffd700 1 -- 172.21.15.105:0/3163629978 learned_addr learned my addr 172.21.15.105:0/3163629978 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f03137fe700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314108790 0x7f0314101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 -- 172.21.15.105:0/3163629978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 msgr2=0x7f0314102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 -- 172.21.15.105:0/3163629978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314108790 msgr2=0x7f0314101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314108790 0x7f0314101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 -- 172.21.15.105:0/3163629978 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03141022d0 con 0x7f0314105e60 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f0312ffd700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.090+0000 7f030a7fc700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 0x7f03141014e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c970 tx=0x7f02fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.091+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00e880 con 0x7f0314105e60 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.091+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02fc00f450 con 0x7f0314105e60 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.091+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03141a2d10 con 0x7f0314105e60 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.091+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc018660 con 0x7f0314105e60 2024-04-22T06:32:25.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.091+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03141a3170 con 0x7f0314105e60 2024-04-22T06:32:25.093 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.092+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0314069730 con 0x7f0314105e60 2024-04-22T06:32:25.093 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.093+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f02fc00f600 con 0x7f0314105e60 2024-04-22T06:32:25.093 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.093+0000 7f0310ff9700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 0x7f03000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:25.093 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.093+0000 7f0312ffd700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 0x7f03000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:25.094 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.094+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f02fc0cf6b0 con 0x7f0314105e60 2024-04-22T06:32:25.094 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.094+0000 7f0312ffd700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 0x7f03000a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0304005950 tx=0x7f030400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:25.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.097+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02fc014030 con 0x7f0314105e60 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi177 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.142 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:25.143 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:24 smithi177 ceph-mon[31346]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:32:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi177 2024-04-22T06:32:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:25.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:24 smithi105 ceph-mon[27916]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:32:25.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi177 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 8544M 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:25.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:25.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:25.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:25.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:25.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:24 smithi112 ceph-mon[31236]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T06:32:25.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:25.309+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0314062e80 con 0x7f03000a3c40 2024-04-22T06:32:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:26 smithi105 ceph-mon[27916]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:26 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:26.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:26 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:26.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:26 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:26.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:26 smithi105 ceph-mon[27916]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:32:26.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:26 smithi177 ceph-mon[31346]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:26.414 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:26 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:26.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:26 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:26.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:26 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:26.415 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:26 smithi177 ceph-mon[31346]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:32:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:26 smithi112 ceph-mon[31236]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:26 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:26 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:26 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:26 smithi112 ceph-mon[31236]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T06:32:27.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:27 smithi112 ceph-mon[31236]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:27.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:27 smithi112 ceph-mon[31236]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T06:32:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:27 smithi177 ceph-mon[31346]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:27 smithi177 ceph-mon[31346]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T06:32:28.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:27 smithi105 ceph-mon[27916]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:28.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:27 smithi105 ceph-mon[27916]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T06:32:28.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T06:32:28.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]: dispatch 2024-04-22T06:32:28.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]': finished 2024-04-22T06:32:28.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:32:28.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:28.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2315610867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]: dispatch 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]': finished 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:28.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2315610867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: osdmap e44: 6 total, 6 up, 6 in 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]: dispatch 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3351754407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6ab8f3-9288-43b7-886b-ce713b206fa1"}]': finished 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2315610867' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:29.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:29 smithi112 ceph-mon[31236]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-22T06:32:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:29 smithi177 ceph-mon[31346]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-22T06:32:30.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:29 smithi105 ceph-mon[27916]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-22T06:32:31.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:31 smithi112 ceph-mon[31236]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-22T06:32:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:31 smithi177 ceph-mon[31346]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-22T06:32:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:31 smithi105 ceph-mon[27916]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-22T06:32:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:33 smithi105 ceph-mon[27916]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:32:33.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:33 smithi177 ceph-mon[31346]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:32:34.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:33 smithi112 ceph-mon[31236]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-22T06:32:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:35 smithi105 ceph-mon[27916]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-22T06:32:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:32:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:35 smithi177 ceph-mon[31346]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-22T06:32:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:32:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:36.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:35 smithi112 ceph-mon[31236]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-22T06:32:36.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T06:32:36.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:36 smithi105 ceph-mon[27916]: Deploying daemon osd.6 on smithi105 2024-04-22T06:32:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:36 smithi177 ceph-mon[31346]: Deploying daemon osd.6 on smithi105 2024-04-22T06:32:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:36 smithi112 ceph-mon[31236]: Deploying daemon osd.6 on smithi105 2024-04-22T06:32:37.884 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:37 smithi105 ceph-mon[27916]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:37.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:37 smithi112 ceph-mon[31236]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:37 smithi177 ceph-mon[31346]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:39 smithi177 ceph-mon[31346]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:40.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:39 smithi105 ceph-mon[27916]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:40.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:39 smithi112 ceph-mon[31236]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:41.843 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:41 smithi105 ceph-mon[27916]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:41.843 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:41.843 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:41.843 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:41 smithi177 ceph-mon[31346]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:41 smithi112 ceph-mon[31236]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:43 smithi177 ceph-mon[31346]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:43 smithi105 ceph-mon[27916]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:43 smithi112 ceph-mon[31236]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:44.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.965+0000 7f0310ff9700 1 -- 172.21.15.105:0/3163629978 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0314062e80 con 0x7f03000a3c40 2024-04-22T06:32:44.965 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 6 on host 'smithi105' 2024-04-22T06:32:44.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 msgr2=0x7f03000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:44.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 0x7f03000a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0304005950 tx=0x7f030400b500 comp rx=0 tx=0).stop 2024-04-22T06:32:44.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 msgr2=0x7f03141014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 0x7f03141014e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c970 tx=0x7f02fc00cc80 comp rx=0 tx=0).stop 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 shutdown_connections 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f03000a3c40 0x7f03000a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0314108790 0x7f0314101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0314103530 0x7f0314102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.967+0000 7f0319431700 1 --2- 172.21.15.105:0/3163629978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0314105e60 0x7f03141014e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.968+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 >> 172.21.15.105:0/3163629978 conn(0x7f03140fcf60 msgr2=0x7f0314103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.968+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 shutdown_connections 2024-04-22T06:32:44.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:32:44.968+0000 7f0319431700 1 -- 172.21.15.105:0/3163629978 wait complete. 2024-04-22T06:32:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:45.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:32:45.565 DEBUG:teuthology.orchestra.run.smithi105:osd.6> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T06:32:45.567 INFO:tasks.cephadm:Deploying osd.7 on smithi112 with /dev/vg_nvme/lv_2... 2024-04-22T06:32:45.567 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:32:45.810 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:32:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:45 smithi105 ceph-mon[27916]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:32:46.214 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:45 smithi112 ceph-mon[31236]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:32:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:45 smithi177 ceph-mon[31346]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T06:32:47.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-mon[27916]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:32:47.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-mon[27916]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T06:32:47.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-mon[27916]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:32:47.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:47.066 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-mon[27916]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:47 smithi112 ceph-mon[31236]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:32:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:47 smithi112 ceph-mon[31236]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T06:32:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:47 smithi112 ceph-mon[31236]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:32:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:47.276 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:47 smithi112 ceph-mon[31236]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:47 smithi177 ceph-mon[31346]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T06:32:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:47 smithi177 ceph-mon[31346]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T06:32:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:47 smithi177 ceph-mon[31346]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:32:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:47 smithi177 ceph-mon[31346]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:47.849 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:32:47.849 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:32:47.849 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-22T06:32:47.850 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-22T06:32:47.850 INFO:teuthology.orchestra.run.smithi112.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116715 s, 898 MB/s 2024-04-22T06:32:47.850 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-22T06:32:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:32:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: osdmap e47: 7 total, 6 up, 7 in 2024-04-22T06:32:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' 2024-04-22T06:32:48.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.169 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:32:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:32:47.905+0000 7fdfec561700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T06:32:48.169 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:32:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:32:48.022+0000 7fdfe3b4b700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:32:48.342 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:32:48.342 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: osdmap e47: 7 total, 6 up, 7 in 2024-04-22T06:32:48.342 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.343 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.343 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' 2024-04-22T06:32:48.343 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:48 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: osdmap e47: 7 total, 6 up, 7 in 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: from='osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953]' entity='osd.6' 2024-04-22T06:32:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:48 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:48.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2024-04-22T06:32:48.899 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi105 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-22T06:32:49.205 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.206 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:49.206 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:49.206 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.206 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:49.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi105 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:49.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:49.420 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.420 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi105 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 5696M 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:32:49.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:50.189 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:50 smithi112 ceph-mon[31236]: osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] boot 2024-04-22T06:32:50.189 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:50 smithi112 ceph-mon[31236]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T06:32:50.189 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 -- 172.21.15.112:0/2428700699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 msgr2=0x7f05e0104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 --2- 172.21.15.112:0/2428700699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0104250 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f05c80099c0 tx=0x7f05c8009cd0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 -- 172.21.15.112:0/2428700699 shutdown_connections 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 --2- 172.21.15.112:0/2428700699 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0104790 0x7f05e010b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 --2- 172.21.15.112:0/2428700699 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0100d70 0x7f05e0101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 --2- 172.21.15.112:0/2428700699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0104250 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 -- 172.21.15.112:0/2428700699 >> 172.21.15.112:0/2428700699 conn(0x7f05e00fc6d0 msgr2=0x7f05e00feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 -- 172.21.15.112:0/2428700699 shutdown_connections 2024-04-22T06:32:50.385 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 -- 172.21.15.112:0/2428700699 wait complete. 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.384+0000 7f05e5640700 1 Processor -- start 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 -- start start 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0100d70 0x7f05e0198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 0x7f05e01a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010e670 con 0x7f05e0104790 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010e4f0 con 0x7f05e0103df0 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05e5640700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010e7f0 con 0x7f05e0100d70 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:45326/0 (socket says 172.21.15.112:45326) 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 -- 172.21.15.112:0/3392083491 learned_addr learned my addr 172.21.15.112:0/3392083491 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05deffd700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0100d70 0x7f05e0198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05df7fe700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 0x7f05e01a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:50.386 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 -- 172.21.15.112:0/3392083491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0100d70 msgr2=0x7f05e0198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0100d70 0x7f05e0198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 -- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 msgr2=0x7f05e01a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 0x7f05e01a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 -- 172.21.15.112:0/3392083491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e01a0950 con 0x7f05e0103df0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05df7fe700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 0x7f05e01a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.385+0000 7f05de7fc700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009f50 tx=0x7f05c8003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.386+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8027070 con 0x7f05e0103df0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.386+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e01a0be0 con 0x7f05e0103df0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.386+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c8022470 con 0x7f05e0103df0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.386+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c800f6c0 con 0x7f05e0103df0 2024-04-22T06:32:50.387 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.386+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e01a1040 con 0x7f05e0103df0 2024-04-22T06:32:50.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.387+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e0069730 con 0x7f05e0103df0 2024-04-22T06:32:50.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.388+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f05c802a070 con 0x7f05e0103df0 2024-04-22T06:32:50.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.388+0000 7f05d7fff700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 0x7f05cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:32:50.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.389+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7f05c80d0820 con 0x7f05e0103df0 2024-04-22T06:32:50.388 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.389+0000 7f05deffd700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 0x7f05cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:32:50.389 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.389+0000 7f05deffd700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 0x7f05cc0a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05e019a1e0 tx=0x7f05d00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:32:50.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.393+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05c805eff0 con 0x7f05e0103df0 2024-04-22T06:32:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:50 smithi105 ceph-mon[27916]: osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] boot 2024-04-22T06:32:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:50 smithi105 ceph-mon[27916]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T06:32:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:50 smithi177 ceph-mon[31346]: osd.6 [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] boot 2024-04-22T06:32:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:50 smithi177 ceph-mon[31346]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T06:32:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:32:50.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:32:50.599+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f05e0062e80 con 0x7f05cc0a3c40 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:51.321 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:51.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T06:32:51.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:51.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:51.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:51.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:51.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:32:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:32:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/3835803068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]': finished 2024-04-22T06:32:53.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:32:53.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:53 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/3835803068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]': finished 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:32:53.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:53 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/3835803068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]: dispatch 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d324cb7-ac5a-44a5-b33f-4fb140559f77"}]': finished 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T06:32:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:53 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:32:54.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:54 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/1262206829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:54.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:54 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/1262206829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:54.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:54 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/1262206829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:32:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:55 smithi105 ceph-mon[27916]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:55 smithi112 ceph-mon[31236]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:55 smithi177 ceph-mon[31346]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:57.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:57 smithi112 ceph-mon[31236]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:57 smithi105 ceph-mon[27916]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:57 smithi177 ceph-mon[31346]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:59.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:32:59 smithi105 ceph-mon[27916]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:59.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:32:59 smithi112 ceph-mon[31236]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:32:59.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:32:59 smithi177 ceph-mon[31346]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:00.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:33:00.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:00.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:33:00.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:00.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T06:33:00.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:01.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:01 smithi105 ceph-mon[27916]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:01.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:01 smithi105 ceph-mon[27916]: Deploying daemon osd.7 on smithi112 2024-04-22T06:33:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:01 smithi112 ceph-mon[31236]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:01 smithi112 ceph-mon[31236]: Deploying daemon osd.7 on smithi112 2024-04-22T06:33:01.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:01 smithi177 ceph-mon[31346]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:01 smithi177 ceph-mon[31346]: Deploying daemon osd.7 on smithi112 2024-04-22T06:33:03.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:03 smithi105 ceph-mon[27916]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:03.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:03 smithi112 ceph-mon[31236]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:03 smithi177 ceph-mon[31346]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:05.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:05 smithi105 ceph-mon[27916]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:05.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:05 smithi112 ceph-mon[31236]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:05 smithi177 ceph-mon[31346]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:07.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:07 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:07.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:07 smithi105 ceph-mon[27916]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:07.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:07 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:07.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:07 smithi112 ceph-mon[31236]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:07.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:07.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:07 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:07.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:07 smithi177 ceph-mon[31346]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:09.364 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.364+0000 7f05d7fff700 1 -- 172.21.15.112:0/3392083491 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f05e0062e80 con 0x7f05cc0a3c40 2024-04-22T06:33:09.364 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 7 on host 'smithi112' 2024-04-22T06:33:09.365 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 msgr2=0x7f05cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 0x7f05cc0a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f05e019a1e0 tx=0x7f05d00094f0 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 msgr2=0x7f05e0199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05c8009f50 tx=0x7f05c8003a10 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 shutdown_connections 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05cc0a3c40 0x7f05cc0a6100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05e0104790 0x7f05e01a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05e0100d70 0x7f05e0198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 --2- 172.21.15.112:0/3392083491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05e0103df0 0x7f05e0199240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.366+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 >> 172.21.15.112:0/3392083491 conn(0x7f05e00fc6d0 msgr2=0x7f05e0107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.367+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 shutdown_connections 2024-04-22T06:33:09.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:33:09.367+0000 7f05e5640700 1 -- 172.21.15.112:0/3392083491 wait complete. 2024-04-22T06:33:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:09 smithi105 ceph-mon[27916]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:09.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:09 smithi112 ceph-mon[31236]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:09.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:09.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:09.961 DEBUG:teuthology.orchestra.run.smithi112:osd.7> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T06:33:09.964 INFO:tasks.cephadm:Deploying osd.8 on smithi177 with /dev/vg_nvme/lv_2... 2024-04-22T06:33:09.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T06:33:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:09 smithi177 ceph-mon[31346]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.207 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:33:10.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:10 smithi112 ceph-mon[31236]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.914 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:10 smithi112 ceph-mon[31236]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.914 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.914 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:10 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:10 smithi105 ceph-mon[27916]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:10 smithi105 ceph-mon[27916]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:10 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:10 smithi177 ceph-mon[31346]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:10 smithi177 ceph-mon[31346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T06:33:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:10 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:11 smithi177 ceph-mon[31346]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:33:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:33:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T06:33:11.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:11.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:11 smithi105 ceph-mon[27916]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:33:11.931 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:11.931 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T06:33:11.931 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: from='osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.931 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T06:33:11.931 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:11.932 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:33:11.932 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-mon[31236]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:33:11.932 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:33:11.651+0000 7fb1704f6700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T06:33:11.932 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:33:11 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:33:11.663+0000 7fb16c2e9700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129291 s, 811 MB/s 2024-04-22T06:33:12.026 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-22T06:33:12.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-04-22T06:33:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:12 smithi105 ceph-mon[27916]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T06:33:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:12.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:12 smithi112 ceph-mon[31236]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T06:33:12.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:12.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:12.974 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:33:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:12 smithi177 ceph-mon[31346]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T06:33:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] boot 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:13 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] boot 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:13 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: osd.7 [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] boot 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T06:33:13.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:13.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:13 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:14.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 -- 172.21.15.177:0/3430996131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318104650 msgr2=0x7fa31810af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:14.452 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/3430996131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318104650 0x7fa31810af00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa31000b0a0 tx=0x7fa31000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 -- 172.21.15.177:0/3430996131 shutdown_connections 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/3430996131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318104650 0x7fa31810af00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/3430996131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318102ab0 0x7fa318102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/3430996131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318103cb0 0x7fa318104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 -- 172.21.15.177:0/3430996131 >> 172.21.15.177:0/3430996131 conn(0x7fa3180fe050 msgr2=0x7fa318100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 -- 172.21.15.177:0/3430996131 shutdown_connections 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.451+0000 7fa31cbc8700 1 -- 172.21.15.177:0/3430996131 wait complete. 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 Processor -- start 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 -- start start 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 0x7fa3180767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:14.453 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318104650 0x7fa318075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810e530 con 0x7fa318104650 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810e3b0 con 0x7fa318102ab0 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31cbc8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa31810e6b0 con 0x7fa318103cb0 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa315d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa315d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:52504/0 (socket says 172.21.15.177:52504) 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa315d9b700 1 -- 172.21.15.177:0/2033459931 learned_addr learned my addr 172.21.15.177:0/2033459931 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa316d9d700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318104650 0x7fa318075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.452+0000 7fa31659c700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 0x7fa3180767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 -- 172.21.15.177:0/2033459931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 msgr2=0x7fa3180767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 0x7fa3180767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.454 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 -- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318104650 msgr2=0x7fa318075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318104650 0x7fa318075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 -- 172.21.15.177:0/2033459931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa318075c20 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa31659c700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 0x7fa3180767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa315d9b700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa30800c9b0 tx=0x7fa30800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308007b00 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3181a2c10 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa308007ca0 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308018660 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.453+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3181a3150 con 0x7fa318103cb0 2024-04-22T06:33:14.455 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.454+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa318069730 con 0x7fa318103cb0 2024-04-22T06:33:14.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.455+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa30800f450 con 0x7fa318103cb0 2024-04-22T06:33:14.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.455+0000 7fa30f7fe700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 0x7fa3040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:14.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.456+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7fa3080cfad0 con 0x7fa318103cb0 2024-04-22T06:33:14.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.456+0000 7fa31659c700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 0x7fa3040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:14.456 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.456+0000 7fa31659c700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 0x7fa3040a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa318103b10 tx=0x7fa300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:33:14.458 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.459+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa308014030 con 0x7fa318103cb0 2024-04-22T06:33:14.669 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:14.670+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa318062e80 con 0x7fa3040a3c40 2024-04-22T06:33:14.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:14 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi112 2024-04-22T06:33:14.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:14 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-22T06:33:14.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:14 smithi112 ceph-mon[31236]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:33:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:14 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi112 2024-04-22T06:33:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:14 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-22T06:33:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:14 smithi177 ceph-mon[31346]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:33:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:14 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi112 2024-04-22T06:33:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:14 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 5696M 2024-04-22T06:33:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:14 smithi105 ceph-mon[27916]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T06:33:15.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:15.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:15.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:33:15.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:15.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:15.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:15.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:15.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:15.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:33:15.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:16.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:16 smithi112 ceph-mon[31236]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T06:33:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:16 smithi177 ceph-mon[31346]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T06:33:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:16 smithi105 ceph-mon[27916]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/3134688264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]': finished 2024-04-22T06:33:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:33:17.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/3134688264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]': finished 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:33:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T06:33:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/3134688264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]: dispatch 2024-04-22T06:33:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab"}]': finished 2024-04-22T06:33:18.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: osdmap e58: 9 total, 8 up, 9 in 2024-04-22T06:33:18.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:18.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:18 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/474143108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:18.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:18 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/474143108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:19.069 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:18 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/474143108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:19 smithi112 ceph-mon[31236]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:19 smithi177 ceph-mon[31346]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:20.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:19 smithi105 ceph-mon[27916]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:21.853 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:21 smithi177 ceph-mon[31346]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:21.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:21 smithi112 ceph-mon[31236]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:21 smithi105 ceph-mon[27916]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:23.906 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:23 smithi177 ceph-mon[31346]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:23 smithi105 ceph-mon[27916]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:24.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:23 smithi112 ceph-mon[31236]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:24.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:33:24.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:25.110 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:33:25.110 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-22T06:33:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:25.988 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:25 smithi177 ceph-mon[31346]: Deploying daemon osd.8 on smithi177 2024-04-22T06:33:25.988 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:25 smithi177 ceph-mon[31346]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:25 smithi105 ceph-mon[27916]: Deploying daemon osd.8 on smithi177 2024-04-22T06:33:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:25 smithi105 ceph-mon[27916]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:26.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:25 smithi112 ceph-mon[31236]: Deploying daemon osd.8 on smithi177 2024-04-22T06:33:26.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:25 smithi112 ceph-mon[31236]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:27.978 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:27 smithi177 ceph-mon[31346]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:28.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:27 smithi105 ceph-mon[27916]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:28.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:27 smithi112 ceph-mon[31236]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:29 smithi177 ceph-mon[31346]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:30.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:29 smithi105 ceph-mon[27916]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:30.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:29 smithi112 ceph-mon[31236]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:30 smithi177 ceph-mon[31346]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:30 smithi105 ceph-mon[27916]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:31.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:30 smithi112 ceph-mon[31236]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:33.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.292+0000 7fa30f7fe700 1 -- 172.21.15.177:0/2033459931 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa318062e80 con 0x7fa3040a3c40 2024-04-22T06:33:33.292 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 8 on host 'smithi177' 2024-04-22T06:33:33.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 msgr2=0x7fa3040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:33.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 0x7fa3040a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa318103b10 tx=0x7fa300009260 comp rx=0 tx=0).stop 2024-04-22T06:33:33.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 msgr2=0x7fa318074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:33.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa30800c9b0 tx=0x7fa30800ccc0 comp rx=0 tx=0).stop 2024-04-22T06:33:33.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 shutdown_connections 2024-04-22T06:33:33.308 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa3040a3c40 0x7fa3040a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa318102ab0 0x7fa3180767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa318104650 0x7fa318075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 --2- 172.21.15.177:0/2033459931 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa318103cb0 0x7fa318074e30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 >> 172.21.15.177:0/2033459931 conn(0x7fa3180fe050 msgr2=0x7fa318106ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 shutdown_connections 2024-04-22T06:33:33.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:33:33.295+0000 7fa31cbc8700 1 -- 172.21.15.177:0/2033459931 wait complete. 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:33.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:33.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:33.891 DEBUG:teuthology.orchestra.run.smithi177:osd.8> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.8.service 2024-04-22T06:33:33.895 INFO:tasks.cephadm:Deploying osd.9 on smithi105 with /dev/vg_nvme/lv_1... 2024-04-22T06:33:33.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:33:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:34.143 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:33:34.534 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:34 smithi105 ceph-mon[27916]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:34.535 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:34 smithi105 ceph-mon[27916]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:34.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:34 smithi177 ceph-mon[31346]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:34.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:34 smithi177 ceph-mon[31346]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:34.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:34 smithi112 ceph-mon[31236]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:34.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:34 smithi112 ceph-mon[31236]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-22T06:33:35.573 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:33:35.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: osdmap e59: 9 total, 8 up, 9 in 2024-04-22T06:33:35.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:35.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.574 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:35 smithi105 ceph-mon[27916]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:33:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: osdmap e59: 9 total, 8 up, 9 in 2024-04-22T06:33:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:35 smithi177 ceph-mon[31346]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:35.811 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-22T06:33:35.811 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: osdmap e59: 9 total, 8 up, 9 in 2024-04-22T06:33:35.811 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:35.811 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: from='osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.811 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:33:35.812 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:35 smithi112 ceph-mon[31236]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:33:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:33:36.258 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-04-22T06:33:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-04-22T06:33:36.259 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115136 s, 911 MB/s 2024-04-22T06:33:36.259 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-04-22T06:33:36.482 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:33:36.413+0000 7ff0f69d0700 -1 osd.8 0 waiting for initial osdmap 2024-04-22T06:33:36.563 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:33:36.563 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: osdmap e60: 9 total, 8 up, 9 in 2024-04-22T06:33:36.563 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.563 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.564 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: osdmap e61: 9 total, 8 up, 9 in 2024-04-22T06:33:36.564 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.564 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:36 smithi105 ceph-mon[27916]: from='osd.8 ' entity='osd.8' 2024-04-22T06:33:36.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: osdmap e60: 9 total, 8 up, 9 in 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: osdmap e61: 9 total, 8 up, 9 in 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:36 smithi112 ceph-mon[31236]: from='osd.8 ' entity='osd.8' 2024-04-22T06:33:36.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:33:36.561+0000 7ff0ee7bb700 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:33:36.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:33:36.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: osdmap e60: 9 total, 8 up, 9 in 2024-04-22T06:33:36.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: osdmap e61: 9 total, 8 up, 9 in 2024-04-22T06:33:36.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:36.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:36 smithi177 ceph-mon[31346]: from='osd.8 ' entity='osd.8' 2024-04-22T06:33:36.881 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2024-04-22T06:33:37.167 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:33:37.544 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:33:37.544 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:33:37.544 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:37.544 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:37.544 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] boot 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:33:37.545 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] boot 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:33:37.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:33:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: osd.8 [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] boot 2024-04-22T06:33:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: osdmap e62: 9 total, 9 up, 9 in 2024-04-22T06:33:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:33:38.506 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:38 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi177 2024-04-22T06:33:38.507 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:38 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-22T06:33:38.507 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:38 smithi105 ceph-mon[27916]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 -- 172.21.15.105:0/4015248244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 msgr2=0x7fa674101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/4015248244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa674101dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6680099c0 tx=0x7fa668009cd0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 -- 172.21.15.105:0/4015248244 shutdown_connections 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/4015248244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa674109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/4015248244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa674101dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/4015248244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 0x7fa674103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 -- 172.21.15.105:0/4015248244 >> 172.21.15.105:0/4015248244 conn(0x7fa6740fcf40 msgr2=0x7fa6740ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 -- 172.21.15.105:0/4015248244 shutdown_connections 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.734+0000 7fa67b0ea700 1 -- 172.21.15.105:0/4015248244 wait complete. 2024-04-22T06:33:38.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 Processor -- start 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 -- start start 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 0x7fa674103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa6741044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410d430 con 0x7fa6741019a0 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410d2b0 con 0x7fa674103540 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa67b0ea700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa67410d5b0 con 0x7fa674102ba0 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa678e86700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa678e86700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60586/0 (socket says 172.21.15.105:60586) 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa678e86700 1 -- 172.21.15.105:0/3413319500 learned_addr learned my addr 172.21.15.105:0/3413319500 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa679687700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa6741044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.735+0000 7fa673fff700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 0x7fa674103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 -- 172.21.15.105:0/3413319500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 msgr2=0x7fa674103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:38.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 0x7fa674103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 -- 172.21.15.105:0/3413319500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 msgr2=0x7fa6741044c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa6741044c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 -- 172.21.15.105:0/3413319500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa674104d90 con 0x7fa6741019a0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa679687700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa6741044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa678e86700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa668005b40 tx=0x7fa668009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668027070 con 0x7fa6741019a0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6741a2d40 con 0x7fa6741019a0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa66800fc70 con 0x7fa6741019a0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6680173f0 con 0x7fa6741019a0 2024-04-22T06:33:38.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.736+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6741a31a0 con 0x7fa6741019a0 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.737+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6741a3370 con 0x7fa6741019a0 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.738+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa66802a070 con 0x7fa6741019a0 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.739+0000 7fa671ffb700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 0x7fa6640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.739+0000 7fa673fff700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 0x7fa6640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.739+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5858+0+0 (secure 0 0 0) 0x7fa6680d0a00 con 0x7fa6741019a0 2024-04-22T06:33:38.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.739+0000 7fa673fff700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 0x7fa6640a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa660009bf0 tx=0x7fa660009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:33:38.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:38 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi177 2024-04-22T06:33:38.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:38 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-22T06:33:38.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:38 smithi177 ceph-mon[31346]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:33:38.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.743+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa668031080 con 0x7fa6741019a0 2024-04-22T06:33:38.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:38.950+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa67406ae30 con 0x7fa6640a3c40 2024-04-22T06:33:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:38 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi177 2024-04-22T06:33:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:38 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 5696M 2024-04-22T06:33:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:38 smithi112 ceph-mon[31236]: osdmap e63: 9 total, 9 up, 9 in 2024-04-22T06:33:39.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:39 smithi105 ceph-mon[27916]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:39.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:39.795 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:39.795 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:39.795 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:39 smithi105 ceph-mon[27916]: osdmap e64: 9 total, 9 up, 9 in 2024-04-22T06:33:39.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:39 smithi112 ceph-mon[31236]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:39 smithi112 ceph-mon[31236]: osdmap e64: 9 total, 9 up, 9 in 2024-04-22T06:33:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:39 smithi177 ceph-mon[31346]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:33:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:33:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:39 smithi177 ceph-mon[31346]: osdmap e64: 9 total, 9 up, 9 in 2024-04-22T06:33:40.575 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:40 smithi105 ceph-mon[27916]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:40.576 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:40 smithi105 ceph-mon[27916]: osdmap e65: 9 total, 9 up, 9 in 2024-04-22T06:33:40.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:40 smithi112 ceph-mon[31236]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:40.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:40 smithi112 ceph-mon[31236]: osdmap e65: 9 total, 9 up, 9 in 2024-04-22T06:33:40.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:40 smithi177 ceph-mon[31346]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:33:40.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:40 smithi177 ceph-mon[31346]: osdmap e65: 9 total, 9 up, 9 in 2024-04-22T06:33:41.890 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:41 smithi105 ceph-mon[27916]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:41.891 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:41 smithi105 ceph-mon[27916]: osdmap e66: 9 total, 9 up, 9 in 2024-04-22T06:33:41.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:41 smithi112 ceph-mon[31236]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:41.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:41 smithi112 ceph-mon[31236]: osdmap e66: 9 total, 9 up, 9 in 2024-04-22T06:33:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:41 smithi177 ceph-mon[31346]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:41 smithi177 ceph-mon[31346]: osdmap e66: 9 total, 9 up, 9 in 2024-04-22T06:33:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]: dispatch 2024-04-22T06:33:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]': finished 2024-04-22T06:33:42.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:42 smithi105 ceph-mon[27916]: osdmap e67: 10 total, 9 up, 10 in 2024-04-22T06:33:42.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:42 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:33:42.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2297494999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:42.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]: dispatch 2024-04-22T06:33:42.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]': finished 2024-04-22T06:33:42.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:42 smithi112 ceph-mon[31236]: osdmap e67: 10 total, 9 up, 10 in 2024-04-22T06:33:42.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:42 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:33:42.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2297494999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:42.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]: dispatch 2024-04-22T06:33:42.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1964041043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ed862a3-bff9-49d5-abfe-ff784bbad0aa"}]': finished 2024-04-22T06:33:42.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:42 smithi177 ceph-mon[31346]: osdmap e67: 10 total, 9 up, 10 in 2024-04-22T06:33:42.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:42 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:33:42.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2297494999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:33:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:43 smithi105 ceph-mon[27916]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:43.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:43 smithi112 ceph-mon[31236]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:43.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:43 smithi177 ceph-mon[31346]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:45.896 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:45 smithi105 ceph-mon[27916]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-22T06:33:45.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:45 smithi112 ceph-mon[31236]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-22T06:33:45.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:45 smithi177 ceph-mon[31346]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-22T06:33:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:47 smithi105 ceph-mon[27916]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:33:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:47 smithi112 ceph-mon[31236]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:33:47.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:47 smithi177 ceph-mon[31346]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:33:49.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:49 smithi105 ceph-mon[27916]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:33:49.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:33:49.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:49.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:49 smithi112 ceph-mon[31236]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:33:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:33:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:49 smithi177 ceph-mon[31346]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:33:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-22T06:33:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:33:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:50 smithi105 ceph-mon[27916]: Deploying daemon osd.9 on smithi105 2024-04-22T06:33:50.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:50 smithi112 ceph-mon[31236]: Deploying daemon osd.9 on smithi105 2024-04-22T06:33:50.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:50 smithi177 ceph-mon[31346]: Deploying daemon osd.9 on smithi105 2024-04-22T06:33:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:51 smithi105 ceph-mon[27916]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:33:51.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:51 smithi112 ceph-mon[31236]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:33:51.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:51 smithi177 ceph-mon[31346]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:33:53.782 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:53 smithi105 ceph-mon[27916]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T06:33:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:53 smithi112 ceph-mon[31236]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T06:33:53.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:53 smithi177 ceph-mon[31346]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T06:33:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:55 smithi105 ceph-mon[27916]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:33:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:55 smithi112 ceph-mon[31236]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:33:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:55.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:55 smithi177 ceph-mon[31346]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:33:55.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:33:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:57 smithi105 ceph-mon[27916]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:57 smithi112 ceph-mon[31236]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:57 smithi177 ceph-mon[31346]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:58.185 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 9 on host 'smithi105' 2024-04-22T06:33:58.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.186+0000 7fa671ffb700 1 -- 172.21.15.105:0/3413319500 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa67406ae30 con 0x7fa6640a3c40 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 msgr2=0x7fa6640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 0x7fa6640a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa660009bf0 tx=0x7fa660009260 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 msgr2=0x7fa6741058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa668005b40 tx=0x7fa668009580 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 shutdown_connections 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa6640a3c40 0x7fa6640a6100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa674103540 0x7fa6741044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa6741019a0 0x7fa6741058e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 --2- 172.21.15.105:0/3413319500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa674102ba0 0x7fa674103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 >> 172.21.15.105:0/3413319500 conn(0x7fa6740fcf40 msgr2=0x7fa67406a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:33:58.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 shutdown_connections 2024-04-22T06:33:58.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:33:58.188+0000 7fa67b0ea700 1 -- 172.21.15.105:0/3413319500 wait complete. 2024-04-22T06:33:58.918 DEBUG:teuthology.orchestra.run.smithi105:osd.9> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.9.service 2024-04-22T06:33:58.920 INFO:tasks.cephadm:Deploying osd.10 on smithi112 with /dev/vg_nvme/lv_1... 2024-04-22T06:33:58.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:33:59.168 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:33:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:33:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:59.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:33:59 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:33:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:33:59 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:33:59.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:33:59 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:00.274 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:00 smithi105 ceph-mon[27916]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:34:00.274 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:00 smithi105 ceph-mon[27916]: osdmap e68: 10 total, 9 up, 10 in 2024-04-22T06:34:00.274 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:00 smithi105 ceph-mon[27916]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:34:00.274 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:00 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:00.362 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:00 smithi112 ceph-mon[31236]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:34:00.362 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:00 smithi112 ceph-mon[31236]: osdmap e68: 10 total, 9 up, 10 in 2024-04-22T06:34:00.362 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:00 smithi112 ceph-mon[31236]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:34:00.362 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:00 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:00.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:00 smithi177 ceph-mon[31346]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-22T06:34:00.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:00 smithi177 ceph-mon[31346]: osdmap e68: 10 total, 9 up, 10 in 2024-04-22T06:34:00.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:00 smithi177 ceph-mon[31346]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-04-22T06:34:00.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:00 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.164 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:34:01.097+0000 7fa975e69700 -1 osd.9 0 waiting for initial osdmap 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115903 s, 905 MB/s 2024-04-22T06:34:01.208 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: osdmap e69: 10 total, 9 up, 10 in 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' 2024-04-22T06:34:01.232 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:01 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:34:01.213+0000 7fa96dc54700 -1 osd.9 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:34:01.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:34:01.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: osdmap e69: 10 total, 9 up, 10 in 2024-04-22T06:34:01.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:01.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' 2024-04-22T06:34:01.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:01 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: osdmap e69: 10 total, 9 up, 10 in 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: from='osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130]' entity='osd.9' 2024-04-22T06:34:01.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:01 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:01.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_1 2024-04-22T06:34:02.126 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:34:02.406 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:02 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:34:02.406 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:02 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:34:02.407 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:02 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:02 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:34:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:02 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:34:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:02 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:02 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:34:02.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:02 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:34:02.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:02 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:03.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] boot 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: osdmap e70: 10 total, 10 up, 10 in 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi105 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] boot 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: osdmap e70: 10 total, 10 up, 10 in 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi105 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: osd.9 [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] boot 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: osdmap e70: 10 total, 10 up, 10 in 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi105 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 4272M 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:03.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 -- 172.21.15.112:0/957067297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 msgr2=0x7fabb4102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 --2- 172.21.15.112:0/957067297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb4102ec0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faba80099c0 tx=0x7faba8009cd0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 -- 172.21.15.112:0/957067297 shutdown_connections 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 --2- 172.21.15.112:0/957067297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb4102ec0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 --2- 172.21.15.112:0/957067297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4104640 0x7fabb410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.687 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 --2- 172.21.15.112:0/957067297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4103ca0 0x7fabb4104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 -- 172.21.15.112:0/957067297 >> 172.21.15.112:0/957067297 conn(0x7fabb40fe040 msgr2=0x7fabb4100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 -- 172.21.15.112:0/957067297 shutdown_connections 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.685+0000 7fabbbedb700 1 -- 172.21.15.112:0/957067297 wait complete. 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 Processor -- start 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 -- start start 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4103ca0 0x7fabb4074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4104640 0x7fabb4075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fabb410e510 con 0x7fabb4103ca0 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fabb410e390 con 0x7fabb4102aa0 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabbbedb700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabb410e690 con 0x7fabb4104640 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabb9c77700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:03.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabb9c77700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59984/0 (socket says 172.21.15.112:59984) 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabb9c77700 1 -- 172.21.15.112:0/1146730388 learned_addr learned my addr 172.21.15.112:0/1146730388 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabb9476700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4103ca0 0x7fabb4074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.686+0000 7fabba478700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4104640 0x7fabb4075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 -- 172.21.15.112:0/1146730388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4104640 msgr2=0x7fabb4075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4104640 0x7fabb4075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 -- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4103ca0 msgr2=0x7fabb4074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4103ca0 0x7fabb4074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 -- 172.21.15.112:0/1146730388 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb4075c20 con 0x7fabb4102aa0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabb9c77700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faba80038c0 tx=0x7faba80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8027070 con 0x7fabb4102aa0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba80040e0 con 0x7fabb4102aa0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb41a2d20 con 0x7fabb4102aa0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8022df0 con 0x7fabb4102aa0 2024-04-22T06:34:03.689 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.687+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb41a3180 con 0x7fabb4102aa0 2024-04-22T06:34:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.688+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb4069730 con 0x7fabb4102aa0 2024-04-22T06:34:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.689+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7faba802a070 con 0x7fabb4102aa0 2024-04-22T06:34:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.689+0000 7faba6ffd700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 0x7faba00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.690+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6261+0+0 (secure 0 0 0) 0x7faba80d0760 con 0x7fabb4102aa0 2024-04-22T06:34:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.690+0000 7fabb9476700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 0x7faba00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.690+0000 7fabb9476700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 0x7faba00a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fabb0005950 tx=0x7fabb000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:03.693 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.694+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faba8013070 con 0x7fabb4102aa0 2024-04-22T06:34:03.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:03.902+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fabb4062e80 con 0x7faba00a3c40 2024-04-22T06:34:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:04 smithi112 ceph-mon[31236]: osdmap e71: 10 total, 10 up, 10 in 2024-04-22T06:34:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:04 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:04 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:04.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:04 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:04 smithi177 ceph-mon[31346]: osdmap e71: 10 total, 10 up, 10 in 2024-04-22T06:34:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:04 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:04 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:04 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:04 smithi105 ceph-mon[27916]: osdmap e71: 10 total, 10 up, 10 in 2024-04-22T06:34:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:04 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:04 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:04 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:05.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:05 smithi112 ceph-mon[31236]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:05.444 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:05 smithi112 ceph-mon[31236]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:05.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:05 smithi177 ceph-mon[31346]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:05.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:05 smithi177 ceph-mon[31346]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:05.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:05 smithi105 ceph-mon[27916]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:05.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:05 smithi105 ceph-mon[27916]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:06.898 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:06 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/746040585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.898 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:06 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.899 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:06 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]': finished 2024-04-22T06:34:06.899 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:06 smithi112 ceph-mon[31236]: osdmap e72: 11 total, 10 up, 11 in 2024-04-22T06:34:06.899 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:06 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/746040585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:06 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:06 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]': finished 2024-04-22T06:34:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:06 smithi105 ceph-mon[27916]: osdmap e72: 11 total, 10 up, 11 in 2024-04-22T06:34:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:06 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/746040585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:06 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]: dispatch 2024-04-22T06:34:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:06 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8cd8511d-79e2-438b-8e64-d3936ea4fc3f"}]': finished 2024-04-22T06:34:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:06 smithi177 ceph-mon[31346]: osdmap e72: 11 total, 10 up, 11 in 2024-04-22T06:34:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:07.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:07 smithi105 ceph-mon[27916]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:07.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:07 smithi105 ceph-mon[27916]: from='client.? 172.21.15.112:0/1705204697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:07.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:07 smithi112 ceph-mon[31236]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:07.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:07 smithi112 ceph-mon[31236]: from='client.? 172.21.15.112:0/1705204697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:07.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:07 smithi177 ceph-mon[31346]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:07.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:07 smithi177 ceph-mon[31346]: from='client.? 172.21.15.112:0/1705204697' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:09 smithi105 ceph-mon[27916]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:09.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:09 smithi112 ceph-mon[31236]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:09 smithi177 ceph-mon[31346]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:11.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:11 smithi105 ceph-mon[27916]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:11.939 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:11 smithi112 ceph-mon[31236]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:11 smithi177 ceph-mon[31346]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:13.866 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:13 smithi112 ceph-mon[31236]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:13.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:13 smithi105 ceph-mon[27916]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:13.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:13 smithi177 ceph-mon[31346]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:14.895 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:34:14.895 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:34:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-22T06:34:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:15.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:15 smithi112 ceph-mon[31236]: Deploying daemon osd.10 on smithi112 2024-04-22T06:34:15.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:15 smithi112 ceph-mon[31236]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:15.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:15 smithi105 ceph-mon[27916]: Deploying daemon osd.10 on smithi112 2024-04-22T06:34:15.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:15 smithi105 ceph-mon[27916]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:15 smithi177 ceph-mon[31346]: Deploying daemon osd.10 on smithi112 2024-04-22T06:34:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:15 smithi177 ceph-mon[31346]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:17.901 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:17 smithi112 ceph-mon[31236]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:17.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:17 smithi105 ceph-mon[27916]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:17 smithi177 ceph-mon[31346]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:19.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:19 smithi105 ceph-mon[27916]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:19 smithi112 ceph-mon[31236]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:19.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:19 smithi177 ceph-mon[31346]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:21.776 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:21 smithi112 ceph-mon[31236]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:21 smithi177 ceph-mon[31346]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:21 smithi105 ceph-mon[27916]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:23.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.021+0000 7faba6ffd700 1 -- 172.21.15.112:0/1146730388 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fabb4062e80 con 0x7faba00a3c40 2024-04-22T06:34:23.019 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 10 on host 'smithi112' 2024-04-22T06:34:23.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 msgr2=0x7faba00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 0x7faba00a6100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fabb0005950 tx=0x7fabb000b410 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 msgr2=0x7fabb40767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faba80038c0 tx=0x7faba80047b0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 shutdown_connections 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faba00a3c40 0x7faba00a6100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabb4102aa0 0x7fabb40767e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabb4104640 0x7fabb4075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 --2- 172.21.15.112:0/1146730388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabb4103ca0 0x7fabb4074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 >> 172.21.15.112:0/1146730388 conn(0x7fabb40fe040 msgr2=0x7fabb4107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 shutdown_connections 2024-04-22T06:34:23.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-04-22T06:34:23.023+0000 7fabbbedb700 1 -- 172.21.15.112:0/1146730388 wait complete. 2024-04-22T06:34:23.588 DEBUG:teuthology.orchestra.run.smithi112:osd.10> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.10.service 2024-04-22T06:34:23.590 INFO:tasks.cephadm:Deploying osd.11 on smithi177 with /dev/vg_nvme/lv_1... 2024-04-22T06:34:23.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T06:34:23.841 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:34:23.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:23 smithi112 ceph-mon[31236]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:23 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:23 smithi112 ceph-mon[31236]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:23 smithi112 ceph-mon[31236]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:23 smithi177 ceph-mon[31346]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:23 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:23 smithi177 ceph-mon[31346]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:23.959 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:23 smithi177 ceph-mon[31346]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:23 smithi105 ceph-mon[27916]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:23 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:23 smithi105 ceph-mon[27916]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:23 smithi105 ceph-mon[27916]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-22T06:34:24.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: osdmap e73: 11 total, 10 up, 11 in 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:24.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:24 smithi112 ceph-mon[31236]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: osdmap e73: 11 total, 10 up, 11 in 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:25.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:24 smithi177 ceph-mon[31346]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: osdmap e73: 11 total, 10 up, 11 in 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:25.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:24 smithi105 ceph-mon[27916]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-04-22T06:34:25.649 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T06:34:25.649 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T06:34:25.649 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-04-22T06:34:25.649 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-04-22T06:34:25.650 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111697 s, 939 MB/s 2024-04-22T06:34:25.650 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-04-22T06:34:25.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-mon[31236]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:25.925 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-mon[31236]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:34:25.925 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-mon[31236]: osdmap e74: 11 total, 10 up, 11 in 2024-04-22T06:34:25.925 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:25.925 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:25 smithi105 ceph-mon[27916]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:25 smithi105 ceph-mon[27916]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:34:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:25 smithi105 ceph-mon[27916]: osdmap e74: 11 total, 10 up, 11 in 2024-04-22T06:34:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:26.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:26.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:34:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:34:25.924+0000 7f0553d90700 -1 osd.10 0 waiting for initial osdmap 2024-04-22T06:34:26.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:34:26.046+0000 7f054bb7b700 -1 osd.10 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:34:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:25 smithi177 ceph-mon[31346]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:25 smithi177 ceph-mon[31346]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-04-22T06:34:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:25 smithi177 ceph-mon[31346]: osdmap e74: 11 total, 10 up, 11 in 2024-04-22T06:34:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:26.350 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-04-22T06:34:26.594 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: from='osd.10 ' entity='osd.10' 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: osdmap e75: 11 total, 10 up, 11 in 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.035 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:26 smithi112 ceph-mon[31236]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: from='osd.10 ' entity='osd.10' 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: osdmap e75: 11 total, 10 up, 11 in 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.210 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:26 smithi177 ceph-mon[31346]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:34:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:34:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: from='osd.10 ' entity='osd.10' 2024-04-22T06:34:27.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: osdmap e75: 11 total, 10 up, 11 in 2024-04-22T06:34:27.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:27.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:26 smithi105 ceph-mon[27916]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-22T06:34:28.054 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.053+0000 7feb66f94700 1 -- 172.21.15.177:0/2128730464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 msgr2=0x7feb60109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.053+0000 7feb66f94700 1 --2- 172.21.15.177:0/2128730464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb60109df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b0a0 tx=0x7feb5800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- 172.21.15.177:0/2128730464 shutdown_connections 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- 172.21.15.177:0/2128730464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb601019a0 0x7feb60101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- 172.21.15.177:0/2128730464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb60109df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- 172.21.15.177:0/2128730464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb60102ba0 0x7feb60103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- 172.21.15.177:0/2128730464 >> 172.21.15.177:0/2128730464 conn(0x7feb600fcf60 msgr2=0x7feb600ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- 172.21.15.177:0/2128730464 shutdown_connections 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- 172.21.15.177:0/2128730464 wait complete. 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 Processor -- start 2024-04-22T06:34:28.055 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- start start 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 0x7feb6006bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb60102ba0 0x7feb6019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d420 con 0x7feb601019a0 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d2a0 con 0x7feb60102ba0 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.054+0000 7feb66f94700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feb6010d5a0 con 0x7feb60103540 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb64d30700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 0x7feb6006bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46380/0 (socket says 172.21.15.177:46380) 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb64d30700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 0x7feb6006bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.177:48678/0 (socket says 172.21.15.177:48678) 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 -- 172.21.15.177:0/1777827258 learned_addr learned my addr 172.21.15.177:0/1777827258 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb5ffff700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb60102ba0 0x7feb6019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 -- 172.21.15.177:0/1777827258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb60102ba0 msgr2=0x7feb6019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:28.056 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb60102ba0 0x7feb6019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 -- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 msgr2=0x7feb6006bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 0x7feb6006bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 -- 172.21.15.177:0/1777827258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb601a29c0 con 0x7feb60103540 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb65531700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b070 tx=0x7feb58004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb58021030 con 0x7feb60103540 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb58003a00 con 0x7feb60103540 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb601a2c50 con 0x7feb60103540 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb580045a0 con 0x7feb60103540 2024-04-22T06:34:28.057 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.055+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb601a3100 con 0x7feb60103540 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.056+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb60069730 con 0x7feb60103540 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.057+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7feb58024070 con 0x7feb60103540 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.058+0000 7feb5dffb700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 0x7feb500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.058+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6723+0+0 (secure 0 0 0) 0x7feb5800f020 con 0x7feb60103540 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.058+0000 7feb64d30700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 0x7feb500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:28.058 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.059+0000 7feb64d30700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 0x7feb500a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb60102a00 tx=0x7feb54007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:28.061 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.062+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb5809dab0 con 0x7feb60103540 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] boot 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:28.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi112 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.269 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:28.271+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7feb60062e80 con 0x7feb500a3c40 2024-04-22T06:34:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] boot 2024-04-22T06:34:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:34:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi112 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: osd.10 [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] boot 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: osdmap e76: 11 total, 11 up, 11 in 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi112 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 4272M 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:28.460 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:28.461 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: osdmap e77: 11 total, 11 up, 11 in 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:29.192 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:28 smithi177 ceph-mon[31346]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:29.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: osdmap e77: 11 total, 11 up, 11 in 2024-04-22T06:34:29.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:29.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:29.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:29.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:29.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:28 smithi105 ceph-mon[27916]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:29.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: osdmap e77: 11 total, 11 up, 11 in 2024-04-22T06:34:29.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:34:29.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T06:34:29.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T06:34:29.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:29.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:28 smithi112 ceph-mon[31236]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:30.149 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:29 smithi177 ceph-mon[31346]: osdmap e78: 11 total, 11 up, 11 in 2024-04-22T06:34:30.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:29 smithi105 ceph-mon[27916]: osdmap e78: 11 total, 11 up, 11 in 2024-04-22T06:34:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:29 smithi112 ceph-mon[31236]: osdmap e78: 11 total, 11 up, 11 in 2024-04-22T06:34:31.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:31 smithi177 ceph-mon[31346]: osdmap e79: 11 total, 11 up, 11 in 2024-04-22T06:34:31.329 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:31 smithi177 ceph-mon[31346]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:31 smithi105 ceph-mon[27916]: osdmap e79: 11 total, 11 up, 11 in 2024-04-22T06:34:31.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:31 smithi105 ceph-mon[27916]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:31.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:31 smithi112 ceph-mon[31236]: osdmap e79: 11 total, 11 up, 11 in 2024-04-22T06:34:31.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:31 smithi112 ceph-mon[31236]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: osdmap e80: 11 total, 11 up, 11 in 2024-04-22T06:34:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/4282795257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]': finished 2024-04-22T06:34:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: osdmap e81: 12 total, 11 up, 12 in 2024-04-22T06:34:32.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:32.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:32 smithi105 ceph-mon[27916]: from='client.? 172.21.15.177:0/532772497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:32.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: osdmap e80: 11 total, 11 up, 11 in 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/4282795257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]': finished 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: osdmap e81: 12 total, 11 up, 12 in 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:32.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:32 smithi112 ceph-mon[31236]: from='client.? 172.21.15.177:0/532772497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: osdmap e80: 11 total, 11 up, 11 in 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/4282795257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]: dispatch 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "747a1047-9993-4f2b-9650-a2cddcbe04ff"}]': finished 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: osdmap e81: 12 total, 11 up, 12 in 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:32 smithi177 ceph-mon[31346]: from='client.? 172.21.15.177:0/532772497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T06:34:33.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:33 smithi105 ceph-mon[27916]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:33.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:33 smithi112 ceph-mon[31236]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:33.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:33 smithi177 ceph-mon[31346]: pgmap v245: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:35 smithi105 ceph-mon[27916]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-22T06:34:35.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:35 smithi112 ceph-mon[31236]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-22T06:34:35.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:35 smithi177 ceph-mon[31346]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-22T06:34:37.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:37 smithi105 ceph-mon[27916]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:34:37.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:37 smithi112 ceph-mon[31236]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:34:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:37 smithi177 ceph-mon[31346]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-22T06:34:38.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:34:38.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:34:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:38.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-22T06:34:38.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:39 smithi105 ceph-mon[27916]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:34:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:39 smithi105 ceph-mon[27916]: Deploying daemon osd.11 on smithi177 2024-04-22T06:34:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:39 smithi112 ceph-mon[31236]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:34:40.401 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:39 smithi112 ceph-mon[31236]: Deploying daemon osd.11 on smithi177 2024-04-22T06:34:40.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:39 smithi177 ceph-mon[31346]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-22T06:34:40.402 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:39 smithi177 ceph-mon[31346]: Deploying daemon osd.11 on smithi177 2024-04-22T06:34:41.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:41 smithi112 ceph-mon[31236]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:34:41.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:41 smithi177 ceph-mon[31346]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:34:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:41 smithi105 ceph-mon[27916]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-22T06:34:43.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:43 smithi112 ceph-mon[31236]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T06:34:43.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:43 smithi177 ceph-mon[31346]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T06:34:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:43 smithi105 ceph-mon[27916]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T06:34:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:44 smithi177 ceph-mon[31346]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:34:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:44 smithi105 ceph-mon[27916]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:34:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:44 smithi112 ceph-mon[31236]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T06:34:47.638 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.639+0000 7feb5dffb700 1 -- 172.21.15.177:0/1777827258 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7feb60062e80 con 0x7feb500a3c40 2024-04-22T06:34:47.638 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 11 on host 'smithi177' 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 msgr2=0x7feb500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 0x7feb500a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb60102a00 tx=0x7feb54007400 comp rx=0 tx=0).stop 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 msgr2=0x7feb601a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b070 tx=0x7feb58004e40 comp rx=0 tx=0).stop 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 shutdown_connections 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7feb500a3c40 0x7feb500a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:47.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feb60103540 0x7feb601a21e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:47.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb601019a0 0x7feb6006bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:47.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 --2- 172.21.15.177:0/1777827258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb60102ba0 0x7feb6019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:47.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 >> 172.21.15.177:0/1777827258 conn(0x7feb600fcf60 msgr2=0x7feb60105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:47.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 shutdown_connections 2024-04-22T06:34:47.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:34:47.641+0000 7feb66f94700 1 -- 172.21.15.177:0/1777827258 wait complete. 2024-04-22T06:34:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:47 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.934 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:47 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:47.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:47 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:34:48.281 DEBUG:teuthology.orchestra.run.smithi177:osd.11> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.11.service 2024-04-22T06:34:48.283 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-22T06:34:48.284 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T06:34:48.532 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:34:48.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:48 smithi105 ceph-mon[27916]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:48.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:48 smithi105 ceph-mon[27916]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:48.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:48 smithi112 ceph-mon[31236]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:48.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:48 smithi112 ceph-mon[31236]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:48.973 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:48 smithi177 ceph-mon[31346]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:48.974 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:48 smithi177 ceph-mon[31346]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-22T06:34:49.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:34:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: osdmap e82: 12 total, 11 up, 12 in 2024-04-22T06:34:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:49 smithi112 ceph-mon[31236]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:49.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:34:49.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: osdmap e82: 12 total, 11 up, 12 in 2024-04-22T06:34:49.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:49.969 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.969 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:49 smithi105 ceph-mon[27916]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: osdmap e82: 12 total, 11 up, 12 in 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: from='osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-mon[31346]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-04-22T06:34:49.991 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:34:49.676+0000 7f80a2069700 -1 osd.11 0 waiting for initial osdmap 2024-04-22T06:34:49.991 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:34:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:34:49.694+0000 7f809de5c700 -1 osd.11 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 -- 172.21.15.105:0/3397643410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080104670 msgr2=0x7f008010af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 --2- 172.21.15.105:0/3397643410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080104670 0x7f008010af20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f007800b0d0 tx=0x7f007800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 -- 172.21.15.105:0/3397643410 shutdown_connections 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 --2- 172.21.15.105:0/3397643410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080104670 0x7f008010af20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 --2- 172.21.15.105:0/3397643410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080102ad0 0x7f0080102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 --2- 172.21.15.105:0/3397643410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080103cd0 0x7f0080104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 -- 172.21.15.105:0/3397643410 >> 172.21.15.105:0/3397643410 conn(0x7f00800fe050 msgr2=0x7f00801004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:50.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 -- 172.21.15.105:0/3397643410 shutdown_connections 2024-04-22T06:34:50.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.334+0000 7f0084adc700 1 -- 172.21.15.105:0/3397643410 wait complete. 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 Processor -- start 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 -- start start 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080103cd0 0x7f0080199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 0x7f00801a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f008010e660 con 0x7f0080102ad0 2024-04-22T06:34:50.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f008010e4e0 con 0x7f0080104670 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f0084adc700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f008010e7e0 con 0x7f0080103cd0 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f007e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f007e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51062/0 (socket says 172.21.15.105:51062) 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.335+0000 7f007e59c700 1 -- 172.21.15.105:0/917396919 learned_addr learned my addr 172.21.15.105:0/917396919 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007dd9b700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080103cd0 0x7f0080199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007ed9d700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 0x7f00801a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 -- 172.21.15.105:0/917396919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080103cd0 msgr2=0x7f0080199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080103cd0 0x7f0080199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 -- 172.21.15.105:0/917396919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 msgr2=0x7f00801a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 0x7f00801a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 -- 172.21.15.105:0/917396919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00801a0940 con 0x7f0080102ad0 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007ed9d700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 0x7f00801a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:34:50.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f007e59c700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f007000b500 tx=0x7f007000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070015020 con 0x7f0080102ad0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00801a0c30 con 0x7f0080102ad0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0070011ce0 con 0x7f0080102ad0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0070010d20 con 0x7f0080102ad0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.336+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00801a1120 con 0x7f0080102ad0 2024-04-22T06:34:50.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.337+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0080069730 con 0x7f0080102ad0 2024-04-22T06:34:50.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.339+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0070018020 con 0x7f0080102ad0 2024-04-22T06:34:50.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.340+0000 7f006f7fe700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 0x7f00680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:50.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.340+0000 7f007dd9b700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 0x7f00680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:50.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.340+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 6847+0+0 (secure 0 0 0) 0x7f00700cf5e0 con 0x7f0080102ad0 2024-04-22T06:34:50.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.340+0000 7f007dd9b700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 0x7f00680a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0074009a10 tx=0x7f0074009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:50.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.345+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f007001f020 con 0x7f0080102ad0 2024-04-22T06:34:50.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.551+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f008002e190 con 0x7f0080102ad0 2024-04-22T06:34:50.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.551+0000 7f006f7fe700 1 -- 172.21.15.105:0/917396919 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f007009c580 con 0x7f0080102ad0 2024-04-22T06:34:50.550 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:34:50.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 msgr2=0x7f00680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:50.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 0x7f00680a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0074009a10 tx=0x7f0074009190 comp rx=0 tx=0).stop 2024-04-22T06:34:50.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 msgr2=0x7f0080198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:50.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f007000b500 tx=0x7f007000b810 comp rx=0 tx=0).stop 2024-04-22T06:34:50.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 shutdown_connections 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f00680a3c40 0x7f00680a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0080104670 0x7f00801a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0080102ad0 0x7f0080198cf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 --2- 172.21.15.105:0/917396919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0080103cd0 0x7f0080199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 >> 172.21.15.105:0/917396919 conn(0x7f00800fe050 msgr2=0x7f0080107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 shutdown_connections 2024-04-22T06:34:50.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:50.553+0000 7f0084adc700 1 -- 172.21.15.105:0/917396919 wait complete. 2024-04-22T06:34:50.891 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:50 smithi177 ceph-mon[31346]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:34:50.891 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:50 smithi177 ceph-mon[31346]: osdmap e83: 12 total, 11 up, 12 in 2024-04-22T06:34:50.891 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.891 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:50 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.891 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/917396919' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:50 smithi105 ceph-mon[27916]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:34:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:50 smithi105 ceph-mon[27916]: osdmap e83: 12 total, 11 up, 12 in 2024-04-22T06:34:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:50 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/917396919' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:50.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:50 smithi112 ceph-mon[31236]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-04-22T06:34:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:50 smithi112 ceph-mon[31236]: osdmap e83: 12 total, 11 up, 12 in 2024-04-22T06:34:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:50 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/917396919' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:51.245 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":83,"num_osds":12,"num_up_osds":11,"osd_up_since":1713767666,"num_in_osds":12,"osd_in_since":1713767671,"num_remapped_pgs":0} 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: purged_snaps scrub starts 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: purged_snaps scrub ok 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] boot 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:51.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:51.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:51 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: purged_snaps scrub starts 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: purged_snaps scrub ok 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] boot 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:34:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:51.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:51 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: purged_snaps scrub starts 2024-04-22T06:34:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: purged_snaps scrub ok 2024-04-22T06:34:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-22T06:34:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: osd.11 [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] boot 2024-04-22T06:34:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: osdmap e84: 12 total, 12 up, 12 in 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:34:52.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:51 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:34:52.247 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T06:34:52.490 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:34:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:52 smithi105 ceph-mon[27916]: Detected new or changed devices on smithi177 2024-04-22T06:34:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:52 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-22T06:34:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:52 smithi105 ceph-mon[27916]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:34:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:52 smithi112 ceph-mon[31236]: Detected new or changed devices on smithi177 2024-04-22T06:34:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:52 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-22T06:34:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:52 smithi112 ceph-mon[31236]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:52 smithi177 ceph-mon[31346]: Detected new or changed devices on smithi177 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:52 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 4272M 2024-04-22T06:34:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:52 smithi177 ceph-mon[31346]: osdmap e85: 12 total, 12 up, 12 in 2024-04-22T06:34:53.877 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:53 smithi105 ceph-mon[27916]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:53 smithi112 ceph-mon[31236]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:53.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:53 smithi177 ceph-mon[31346]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:54.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 -- 172.21.15.105:0/3843806269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc1019a0 msgr2=0x7fd1fc101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 --2- 172.21.15.105:0/3843806269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc101dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec0099c0 tx=0x7fd1ec009cd0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 -- 172.21.15.105:0/3843806269 shutdown_connections 2024-04-22T06:34:54.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 --2- 172.21.15.105:0/3843806269 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 --2- 172.21.15.105:0/3843806269 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc101dc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 --2- 172.21.15.105:0/3843806269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 -- 172.21.15.105:0/3843806269 >> 172.21.15.105:0/3843806269 conn(0x7fd1fc0fcf40 msgr2=0x7fd1fc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 -- 172.21.15.105:0/3843806269 shutdown_connections 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.079+0000 7fd203c11700 1 -- 172.21.15.105:0/3843806269 wait complete. 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 Processor -- start 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 -- start start 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc06b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc1a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10d490 con 0x7fd1fc102ba0 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10d310 con 0x7fd1fc1019a0 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd203c11700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10d610 con 0x7fd1fc103540 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51076/0 (socket says 172.21.15.105:51076) 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 -- 172.21.15.105:0/2214860982 learned_addr learned my addr 172.21.15.105:0/2214860982 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2019ad700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2021ae700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc1a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 -- 172.21.15.105:0/2214860982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 msgr2=0x7fd1fc1a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc1a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 -- 172.21.15.105:0/2214860982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc1019a0 msgr2=0x7fd1fc06b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc06b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2011ac700 1 -- 172.21.15.105:0/2214860982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc1a0950 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.080+0000 7fd2021ae700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc1a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd2011ac700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1f800d950 tx=0x7fd1f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8016070 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc1a0c40 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f8009df0 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800f3c0 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.081+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc1a1180 con 0x7fd1fc102ba0 2024-04-22T06:34:54.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.082+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc069730 con 0x7fd1fc102ba0 2024-04-22T06:34:54.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.083+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fd1f8019070 con 0x7fd1fc102ba0 2024-04-22T06:34:54.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.084+0000 7fd1f2ffd700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:54.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.084+0000 7fd2019ad700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:54.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.084+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fd1f80cfb10 con 0x7fd1fc102ba0 2024-04-22T06:34:54.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.084+0000 7fd2019ad700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 0x7fd1e80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec006010 tx=0x7fd1ec003720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:54.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.087+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd1f809c990 con 0x7fd1fc102ba0 2024-04-22T06:34:54.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.287+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd1fc02e050 con 0x7fd1fc102ba0 2024-04-22T06:34:54.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.287+0000 7fd1f2ffd700 1 -- 172.21.15.105:0/2214860982 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v85) ==== 74+0+133 (secure 0 0 0) 0x7fd1f809c290 con 0x7fd1fc102ba0 2024-04-22T06:34:54.286 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:34:54.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 msgr2=0x7fd1e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 0x7fd1e80a6100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec006010 tx=0x7fd1ec003720 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 msgr2=0x7fd1fc19da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1f800d950 tx=0x7fd1f800dc60 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 shutdown_connections 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1fc1019a0 0x7fd1fc06b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1fc103540 0x7fd1fc1a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 --2- 172.21.15.105:0/2214860982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1fc102ba0 0x7fd1fc19da40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 >> 172.21.15.105:0/2214860982 conn(0x7fd1fc0fcf40 msgr2=0x7fd1fc105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 shutdown_connections 2024-04-22T06:34:54.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:54.290+0000 7fd203c11700 1 -- 172.21.15.105:0/2214860982 wait complete. 2024-04-22T06:34:54.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:54 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2214860982' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:54.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:54 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2214860982' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:54.960 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":85,"num_osds":12,"num_up_osds":12,"osd_up_since":1713767690,"num_in_osds":12,"osd_in_since":1713767671,"num_remapped_pgs":0} 2024-04-22T06:34:54.960 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:34:54.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:54 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2214860982' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T06:34:55.194 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:34:55.914 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:55 smithi105 ceph-mon[27916]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:55 smithi112 ceph-mon[31236]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:55 smithi177 ceph-mon[31346]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:56.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 -- 172.21.15.105:0/1992601679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f41011f0 msgr2=0x7f59f41015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/1992601679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f41011f0 0x7f59f41015d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f59e40099c0 tx=0x7f59e4009cd0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 -- 172.21.15.105:0/1992601679 shutdown_connections 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/1992601679 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f4107e10 0x7f59f410f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/1992601679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f41011f0 0x7f59f41015d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/1992601679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 -- 172.21.15.105:0/1992601679 >> 172.21.15.105:0/1992601679 conn(0x7f59f4077f40 msgr2=0x7f59f40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:56.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 -- 172.21.15.105:0/1992601679 shutdown_connections 2024-04-22T06:34:56.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.788+0000 7f59f9f9d700 1 -- 172.21.15.105:0/1992601679 wait complete. 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 Processor -- start 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 -- start start 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112c80 con 0x7f59f4107e10 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112b00 con 0x7f59f41011f0 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f9f9d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59f4112e00 con 0x7f59f4101ba0 2024-04-22T06:34:56.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:48824/0 (socket says 172.21.15.105:48824) 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34388/0 (socket says 172.21.15.105:34388) 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f37fe700 1 -- 172.21.15.105:0/763431798 learned_addr learned my addr 172.21.15.105:0/763431798 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.789+0000 7f59f2ffd700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 -- 172.21.15.105:0/763431798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 msgr2=0x7f59f41996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 -- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 msgr2=0x7f59f419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 -- 172.21.15.105:0/763431798 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f419e1a0 con 0x7f59f41011f0 2024-04-22T06:34:56.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f2ffd700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f3fff700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f37fe700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e40045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4026070 con 0x7f59f41011f0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59e400f460 con 0x7f59f41011f0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f419e430 con 0x7f59f41011f0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e4009e60 con 0x7f59f41011f0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.790+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f41a5d10 con 0x7f59f41011f0 2024-04-22T06:34:56.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.791+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f410b050 con 0x7f59f41011f0 2024-04-22T06:34:56.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.792+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f59e4029070 con 0x7f59f41011f0 2024-04-22T06:34:56.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.793+0000 7f59f0ff9700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:56.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.793+0000 7f59f2ffd700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:56.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.793+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f59e40d07f0 con 0x7f59f41011f0 2024-04-22T06:34:56.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.793+0000 7f59f2ffd700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59f4105610 tx=0x7f59e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:56.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.796+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59e4013070 con 0x7f59f41011f0 2024-04-22T06:34:56.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.997+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59f419a4f0 con 0x7f59f41011f0 2024-04-22T06:34:56.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:56.999+0000 7f59f0ff9700 1 -- 172.21.15.105:0/763431798 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+18502 (secure 0 0 0) 0x7f59e409d6e0 con 0x7f59f41011f0 2024-04-22T06:34:56.998 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:34:56.998 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":85,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:34:51.679181+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:34:56.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 msgr2=0x7f59dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59f4105610 tx=0x7f59e8009260 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 msgr2=0x7f59f4199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59e4006010 tx=0x7f59e40045e0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 shutdown_connections 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59f41011f0 0x7f59f4199190 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59f4107e10 0x7f59f419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 --2- 172.21.15.105:0/763431798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59f4101ba0 0x7f59f41996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 >> 172.21.15.105:0/763431798 conn(0x7f59f4077f40 msgr2=0x7f59f40ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 shutdown_connections 2024-04-22T06:34:57.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:57.001+0000 7f59f9f9d700 1 -- 172.21.15.105:0/763431798 wait complete. 2024-04-22T06:34:57.651 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:34:57.652 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T06:34:57.886 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:34:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:57 smithi105 ceph-mon[27916]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:57 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/763431798' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:34:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:57 smithi112 ceph-mon[31236]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:57 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/763431798' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:34:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:57 smithi177 ceph-mon[31346]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:57 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/763431798' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:34:59.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 -- 172.21.15.105:0/1813212926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 msgr2=0x7f7e88101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 --2- 172.21.15.105:0/1813212926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e88101dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009990 tx=0x7f7e84009ca0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 -- 172.21.15.105:0/1813212926 shutdown_connections 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 --2- 172.21.15.105:0/1813212926 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e88107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 --2- 172.21.15.105:0/1813212926 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 0x7f7e8810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 --2- 172.21.15.105:0/1813212926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e88101dc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 -- 172.21.15.105:0/1813212926 >> 172.21.15.105:0/1813212926 conn(0x7f7e880788f0 msgr2=0x7f7e88078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 -- 172.21.15.105:0/1813212926 shutdown_connections 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.387+0000 7f7e8f448700 1 -- 172.21.15.105:0/1813212926 wait complete. 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 Processor -- start 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 -- start start 2024-04-22T06:34:59.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 0x7f7e88199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e8819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88111b50 con 0x7f7e88101940 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881119d0 con 0x7f7e88107990 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8f448700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7e88111cd0 con 0x7f7e88102300 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8d1e4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8d1e4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34408/0 (socket says 172.21.15.105:34408) 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8d1e4700 1 -- 172.21.15.105:0/4062207939 learned_addr learned my addr 172.21.15.105:0/4062207939 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8c9e3700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 0x7f7e88199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.388+0000 7f7e8d9e5700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e8819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 -- 172.21.15.105:0/4062207939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 msgr2=0x7f7e88199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 0x7f7e88199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 -- 172.21.15.105:0/4062207939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 msgr2=0x7f7e8819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e8819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 -- 172.21.15.105:0/4062207939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8819e200 con 0x7f7e88101940 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d9e5700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e8819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8d1e4700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800d950 tx=0x7f7e7800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78016070 con 0x7f7e88101940 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8819e4f0 con 0x7f7e88101940 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e78009df0 con 0x7f7e88101940 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7800f3c0 con 0x7f7e88101940 2024-04-22T06:34:59.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.389+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e881a5d70 con 0x7f7e88101940 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.390+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e88069730 con 0x7f7e88101940 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.391+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7e78019070 con 0x7f7e88101940 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.392+0000 7f7e7e7fc700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 0x7f7e740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.392+0000 7f7e8c9e3700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 0x7f7e740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.392+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f7e780cfbd0 con 0x7f7e88101940 2024-04-22T06:34:59.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.392+0000 7f7e8c9e3700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 0x7f7e740a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400b5c0 tx=0x7f7e84005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:34:59.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.396+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e7809c9b0 con 0x7f7e88101940 2024-04-22T06:34:59.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.596+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e8819a550 con 0x7f7e88101940 2024-04-22T06:34:59.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.596+0000 7f7e7e7fc700 1 -- 172.21.15.105:0/4062207939 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f7e78096e20 con 0x7f7e88101940 2024-04-22T06:34:59.595 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:34:59.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 msgr2=0x7f7e740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 0x7f7e740a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400b5c0 tx=0x7f7e84005960 comp rx=0 tx=0).stop 2024-04-22T06:34:59.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 msgr2=0x7f7e881991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800d950 tx=0x7f7e7800dc60 comp rx=0 tx=0).stop 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 shutdown_connections 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f7e740a3c40 0x7f7e740a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e88107990 0x7f7e8819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e88101940 0x7f7e881991f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 --2- 172.21.15.105:0/4062207939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7e88102300 0x7f7e88199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:34:59.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 >> 172.21.15.105:0/4062207939 conn(0x7f7e880788f0 msgr2=0x7f7e8806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:34:59.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 shutdown_connections 2024-04-22T06:34:59.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:34:59.598+0000 7f7e8f448700 1 -- 172.21.15.105:0/4062207939 wait complete. 2024-04-22T06:34:59.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:59 smithi105 ceph-mon[27916]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:59.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:34:59 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4062207939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:34:59.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:59 smithi112 ceph-mon[31236]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:59.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:34:59 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4062207939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:34:59.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:59 smithi177 ceph-mon[31346]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:34:59.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:34:59 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4062207939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:00.315 INFO:tasks.cephadm:Adding mds.a on smithi105 2024-04-22T06:35:00.315 INFO:tasks.cephadm:Adding mds.d on smithi105 2024-04-22T06:35:00.315 INFO:tasks.cephadm:Adding mds.g on smithi105 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.j on smithi105 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.b on smithi112 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.e on smithi112 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.h on smithi112 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.k on smithi112 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.c on smithi177 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.f on smithi177 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.i on smithi177 2024-04-22T06:35:00.316 INFO:tasks.cephadm:Adding mds.l on smithi177 2024-04-22T06:35:00.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l' 2024-04-22T06:35:00.564 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:35:01.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:01 smithi177 ceph-mon[31346]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:01 smithi112 ceph-mon[31236]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 -- 172.21.15.177:0/2084994064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8102bc0 msgr2=0x7fbde8103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2084994064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8102bc0 0x7fbde8103020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009990 tx=0x7fbdd4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 -- 172.21.15.177:0/2084994064 shutdown_connections 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2084994064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde81019c0 0x7fbde8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2084994064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde8103560 0x7fbde8109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2084994064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8102bc0 0x7fbde8103020 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 -- 172.21.15.177:0/2084994064 >> 172.21.15.177:0/2084994064 conn(0x7fbde80fcfa0 msgr2=0x7fbde80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 -- 172.21.15.177:0/2084994064 shutdown_connections 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.138+0000 7fbdeecec700 1 -- 172.21.15.177:0/2084994064 wait complete. 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 Processor -- start 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 -- start start 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde81019c0 0x7fbde8105950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde8102bc0 0x7fbde8103fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810d4a0 con 0x7fbde81019c0 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810d320 con 0x7fbde8102bc0 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeecec700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbde810d620 con 0x7fbde8103560 2024-04-22T06:35:02.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbded289700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbded289700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:54188/0 (socket says 172.21.15.177:54188) 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbded289700 1 -- 172.21.15.177:0/2607505742 learned_addr learned my addr 172.21.15.177:0/2607505742 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbde7fff700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde8102bc0 0x7fbde8103fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.139+0000 7fbdeca88700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde81019c0 0x7fbde8105950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 -- 172.21.15.177:0/2607505742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde8102bc0 msgr2=0x7fbde8103fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde8102bc0 0x7fbde8103fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 -- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde81019c0 msgr2=0x7fbde8105950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde81019c0 0x7fbde8105950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 -- 172.21.15.177:0/2607505742 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde8104d90 con 0x7fbde8103560 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbded289700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbde000ada0 tx=0x7fbde000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0017020 con 0x7fbde8103560 2024-04-22T06:35:02.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde81a2de0 con 0x7fbde8103560 2024-04-22T06:35:02.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbde000ce30 con 0x7fbde8103560 2024-04-22T06:35:02.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0005350 con 0x7fbde8103560 2024-04-22T06:35:02.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.140+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde81a3240 con 0x7fbde8103560 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.141+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde8069760 con 0x7fbde8103560 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.142+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fbde000c980 con 0x7fbde8103560 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.143+0000 7fbde5ffb700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 0x7fbdd80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.143+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fbde00cf650 con 0x7fbde8103560 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.143+0000 7fbdeca88700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 0x7fbdd80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:02.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.143+0000 7fbdeca88700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 0x7fbdd80a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbddc005fd0 tx=0x7fbddc00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:02.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.147+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbde8069760 con 0x7fbde8103560 2024-04-22T06:35:02.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:01 smithi105 ceph-mon[27916]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:02.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.357+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}) -- 0x7fbde80ff2b0 con 0x7fbdd80a3c40 2024-04-22T06:35:02.364 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.365+0000 7fbde5ffb700 1 -- 172.21.15.177:0/2607505742 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fbde80ff2b0 con 0x7fbdd80a3c40 2024-04-22T06:35:02.364 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mds.all update... 2024-04-22T06:35:02.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 msgr2=0x7fbdd80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 0x7fbdd80a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbddc005fd0 tx=0x7fbddc00a560 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 msgr2=0x7fbde81044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbde000ada0 tx=0x7fbde000c330 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 shutdown_connections 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbdd80a3c40 0x7fbdd80a6100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbde8103560 0x7fbde81044e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbde81019c0 0x7fbde8105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 --2- 172.21.15.177:0/2607505742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbde8102bc0 0x7fbde8103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 >> 172.21.15.177:0/2607505742 conn(0x7fbde80fcfa0 msgr2=0x7fbde80feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 shutdown_connections 2024-04-22T06:35:02.367 INFO:teuthology.orchestra.run.smithi177.stderr:2024-04-22T06:35:02.368+0000 7fbdeecec700 1 -- 172.21.15.177:0/2607505742 wait complete. 2024-04-22T06:35:02.929 DEBUG:teuthology.orchestra.run.smithi105:mds.a> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service 2024-04-22T06:35:02.932 DEBUG:teuthology.orchestra.run.smithi105:mds.d> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.d.service 2024-04-22T06:35:02.979 DEBUG:teuthology.orchestra.run.smithi105:mds.g> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g.service 2024-04-22T06:35:03.021 DEBUG:teuthology.orchestra.run.smithi105:mds.j> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.j.service 2024-04-22T06:35:03.065 DEBUG:teuthology.orchestra.run.smithi112:mds.b> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.b.service 2024-04-22T06:35:03.067 DEBUG:teuthology.orchestra.run.smithi112:mds.e> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service 2024-04-22T06:35:03.111 DEBUG:teuthology.orchestra.run.smithi112:mds.h> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service 2024-04-22T06:35:03.155 DEBUG:teuthology.orchestra.run.smithi112:mds.k> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service 2024-04-22T06:35:03.197 DEBUG:teuthology.orchestra.run.smithi177:mds.c> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service 2024-04-22T06:35:03.200 DEBUG:teuthology.orchestra.run.smithi177:mds.f> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.f.service 2024-04-22T06:35:03.242 DEBUG:teuthology.orchestra.run.smithi177:mds.i> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service 2024-04-22T06:35:03.285 DEBUG:teuthology.orchestra.run.smithi177:mds.l> sudo journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l.service 2024-04-22T06:35:03.328 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-22T06:35:03.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:35:03.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 -- 172.21.15.105:0/4043049812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50100e70 msgr2=0x7f1a50101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 --2- 172.21.15.105:0/4043049812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50100e70 0x7f1a50101290 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009900 tx=0x7f1a4c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:03.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 -- 172.21.15.105:0/4043049812 shutdown_connections 2024-04-22T06:35:03.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 --2- 172.21.15.105:0/4043049812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102a10 0x7f1a50109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 --2- 172.21.15.105:0/4043049812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50102050 0x7f1a501024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 --2- 172.21.15.105:0/4043049812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50100e70 0x7f1a50101290 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 -- 172.21.15.105:0/4043049812 >> 172.21.15.105:0/4043049812 conn(0x7f1a500fc470 msgr2=0x7f1a500fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 -- 172.21.15.105:0/4043049812 shutdown_connections 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.458+0000 7f1a58025640 1 -- 172.21.15.105:0/4043049812 wait complete. 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 Processor -- start 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 -- start start 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50100e70 0x7f1a5010b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102050 0x7f1a50109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010db40 con 0x7f1a50102a10 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010d9c0 con 0x7f1a50102050 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a58025640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010dcc0 con 0x7f1a50100e70 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34426/0 (socket says 172.21.15.105:34426) 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 -- 172.21.15.105:0/3295859852 learned_addr learned my addr 172.21.15.105:0/3295859852 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a55dc1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50100e70 0x7f1a5010b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a555c0640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102050 0x7f1a50109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 -- 172.21.15.105:0/3295859852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50100e70 msgr2=0x7f1a5010b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50100e70 0x7f1a5010b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 -- 172.21.15.105:0/3295859852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102050 msgr2=0x7f1a50109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102050 0x7f1a50109eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.459+0000 7f1a565c2640 1 -- 172.21.15.105:0/3295859852 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5010ac80 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a565c2640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a4800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48017070 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a501a2320 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a48003e20 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a480106c0 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.460+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a501a2820 con 0x7f1a50102a10 2024-04-22T06:35:03.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.461+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a50108290 con 0x7f1a50102a10 2024-04-22T06:35:03.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.462+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1a48004140 con 0x7f1a50102a10 2024-04-22T06:35:03.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.462+0000 7f1a46ffd640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 0x7f1a3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.463+0000 7f1a55dc1640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 0x7f1a3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.463+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f1a480cfb60 con 0x7f1a50102a10 2024-04-22T06:35:03.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.463+0000 7f1a55dc1640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 0x7f1a3c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c005ab0 tx=0x7f1a4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.467+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a480047e0 con 0x7f1a50102a10 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: Saving service mds.all spec with placement smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:03.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:03 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.648+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a50062b50 con 0x7f1a50102a10 2024-04-22T06:35:03.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.649+0000 7f1a46ffd640 1 -- 172.21.15.105:0/3295859852 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+18502 (secure 0 0 0) 0x7f1a4809c390 con 0x7f1a50102a10 2024-04-22T06:35:03.648 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:03.648 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":85,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:34:51.679181+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 msgr2=0x7f1a3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 0x7f1a3c0a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c005ab0 tx=0x7f1a4c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 msgr2=0x7f1a5010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a48009fd0 tx=0x7f1a4800b260 comp rx=0 tx=0).stop 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 shutdown_connections 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1a3c0a3bb0 0x7f1a3c0a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a50102a10 0x7f1a5010a3f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a50102050 0x7f1a50109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 --2- 172.21.15.105:0/3295859852 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a50100e70 0x7f1a5010b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 >> 172.21.15.105:0/3295859852 conn(0x7f1a500fc470 msgr2=0x7f1a50105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 shutdown_connections 2024-04-22T06:35:03.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.651+0000 7f1a58025640 1 -- 172.21.15.105:0/3295859852 wait complete. 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: Saving service mds.all spec with placement smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.660 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:03.661 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:03.661 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:03 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.662 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:35:03.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:35:03.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:03.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: Saving service mds.all spec with placement smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-22T06:35:03.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:03.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:03 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 -- 172.21.15.105:0/3476879970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc10e560 msgr2=0x7f0ccc110950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/3476879970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc10e560 0x7f0ccc110950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009cd0 tx=0x7f0cc402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 -- 172.21.15.105:0/3476879970 shutdown_connections 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/3476879970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc10e560 0x7f0ccc110950 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/3476879970 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc0749b0 0x7f0ccc074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/3476879970 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc075f90 0x7f0ccc0743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 -- 172.21.15.105:0/3476879970 >> 172.21.15.105:0/3476879970 conn(0x7f0ccc0fd500 msgr2=0x7f0ccc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 -- 172.21.15.105:0/3476879970 shutdown_connections 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.764+0000 7f0cd3f52640 1 -- 172.21.15.105:0/3476879970 wait complete. 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 Processor -- start 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 -- start start 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc0749b0 0x7f0ccc19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 0x7f0ccc1a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc114110 con 0x7f0ccc075f90 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc113f90 con 0x7f0ccc10e560 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd3f52640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc114290 con 0x7f0ccc0749b0 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34454/0 (socket says 172.21.15.105:34454) 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 -- 172.21.15.105:0/1976683288 learned_addr learned my addr 172.21.15.105:0/1976683288 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:03.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd1cee640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc0749b0 0x7f0ccc19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd24ef640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 0x7f0ccc1a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 -- 172.21.15.105:0/1976683288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc0749b0 msgr2=0x7f0ccc19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc0749b0 0x7f0ccc19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 -- 172.21.15.105:0/1976683288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 msgr2=0x7f0ccc1a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 0x7f0ccc1a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 -- 172.21.15.105:0/1976683288 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc1a63f0 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd24ef640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 0x7f0ccc1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.765+0000 7f0cd14ed640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c920 tx=0x7f0cbc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.766+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc007c70 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.766+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cbc007e10 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.766+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc1a66e0 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.766+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc018820 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.766+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc1a6c80 con 0x7f0ccc075f90 2024-04-22T06:35:03.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.767+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ccc104730 con 0x7f0ccc075f90 2024-04-22T06:35:03.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.768+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0cbc00f450 con 0x7f0ccc075f90 2024-04-22T06:35:03.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.769+0000 7f0cc2ffd640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 0x7f0cb80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:03.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.769+0000 7f0cd1cee640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 0x7f0cb80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:03.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.769+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f0cbc0d0390 con 0x7f0ccc075f90 2024-04-22T06:35:03.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.769+0000 7f0cd1cee640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 0x7f0cb80a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009890 tx=0x7f0cc8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:03.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.772+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0cbc014030 con 0x7f0ccc075f90 2024-04-22T06:35:03.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.951+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0ccc02d810 con 0x7f0ccc075f90 2024-04-22T06:35:03.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.951+0000 7f0cc2ffd640 1 -- 172.21.15.105:0/1976683288 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f0cbc09d320 con 0x7f0ccc075f90 2024-04-22T06:35:03.950 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 msgr2=0x7f0cb80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 0x7f0cb80a60f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009890 tx=0x7f0cc8008040 comp rx=0 tx=0).stop 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 msgr2=0x7f0ccc19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c920 tx=0x7f0cbc00cde0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 shutdown_connections 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0cb80a3c30 0x7f0cb80a60f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc10e560 0x7f0ccc1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc075f90 0x7f0ccc19ed10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 --2- 172.21.15.105:0/1976683288 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc0749b0 0x7f0ccc19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:03.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 >> 172.21.15.105:0/1976683288 conn(0x7f0ccc0fd500 msgr2=0x7f0ccc101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:03.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 shutdown_connections 2024-04-22T06:35:03.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:03.953+0000 7f0cd3f52640 1 -- 172.21.15.105:0/1976683288 wait complete. 2024-04-22T06:35:03.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- 172.21.15.105:0/1208204917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 msgr2=0x7f17d4101540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- 172.21.15.105:0/1208204917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4101540 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009900 tx=0x7f17c802bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- 172.21.15.105:0/1208204917 shutdown_connections 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- 172.21.15.105:0/1208204917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 0x7f17d4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- 172.21.15.105:0/1208204917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4101540 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- 172.21.15.105:0/1208204917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d4102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- 172.21.15.105:0/1208204917 >> 172.21.15.105:0/1208204917 conn(0x7f17d40fbb90 msgr2=0x7f17d40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- 172.21.15.105:0/1208204917 shutdown_connections 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- 172.21.15.105:0/1208204917 wait complete. 2024-04-22T06:35:04.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 Processor -- start 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- start start 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 0x7f17d4197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410c9f0 con 0x7f17d41010c0 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.061+0000 7f17db1b6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410c870 con 0x7f17d4101a80 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17db1b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410cb70 con 0x7f17d4102470 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34474/0 (socket says 172.21.15.105:34474) 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 -- 172.21.15.105:0/2835950623 learned_addr learned my addr 172.21.15.105:0/2835950623 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d9753640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d3fff640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 0x7f17d4197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 -- 172.21.15.105:0/2835950623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 msgr2=0x7f17d419e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d419e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 -- 172.21.15.105:0/2835950623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 msgr2=0x7f17d4197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 0x7f17d4197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 -- 172.21.15.105:0/2835950623 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d419ec50 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d9753640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d8f52640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ebb0 tx=0x7f17cc007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc007ca0 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d4100010 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17cc007e40 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0046e0 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.062+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d4100560 con 0x7f17d41010c0 2024-04-22T06:35:04.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.063+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d4069230 con 0x7f17d41010c0 2024-04-22T06:35:04.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.064+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f17cc003970 con 0x7f17d41010c0 2024-04-22T06:35:04.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.065+0000 7f17d1ffb640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 0x7f17bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:04.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.065+0000 7f17d3fff640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 0x7f17bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:04.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.065+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f17cc0d1a10 con 0x7f17d41010c0 2024-04-22T06:35:04.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.065+0000 7f17d3fff640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 0x7f17bc0a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036040 tx=0x7f17c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:04.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.069+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f17cc09e890 con 0x7f17d41010c0 2024-04-22T06:35:04.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.285+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17d4107c10 con 0x7f17d41010c0 2024-04-22T06:35:04.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.286+0000 7f17d1ffb640 1 -- 172.21.15.105:0/2835950623 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f17d4107c10 con 0x7f17d41010c0 2024-04-22T06:35:04.284 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:04.284 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-22T06:35:04.284 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 1 2024-04-22T06:35:04.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 msgr2=0x7f17bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 0x7f17bc0a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f17c8036040 tx=0x7f17c8000bc0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 msgr2=0x7f17d4197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ebb0 tx=0x7f17cc007590 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 shutdown_connections 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f17bc0a3bb0 0x7f17bc0a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f17d4102470 0x7f17d419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d4101a80 0x7f17d4197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 --2- 172.21.15.105:0/2835950623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17d41010c0 0x7f17d4197030 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 >> 172.21.15.105:0/2835950623 conn(0x7f17d40fbb90 msgr2=0x7f17d4105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 shutdown_connections 2024-04-22T06:35:04.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:04.288+0000 7f17db1b6640 1 -- 172.21.15.105:0/2835950623 wait complete. 2024-04-22T06:35:04.574 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:04 smithi177 ceph-mon[31346]: Deploying daemon mds.l on smithi177 2024-04-22T06:35:04.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3295859852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1976683288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:04.575 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2835950623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:04 smithi105 ceph-mon[27916]: Deploying daemon mds.l on smithi177 2024-04-22T06:35:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3295859852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1976683288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2835950623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:04 smithi112 ceph-mon[31236]: Deploying daemon mds.l on smithi177 2024-04-22T06:35:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3295859852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1976683288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2835950623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:04.946 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:04 smithi177 systemd[1]: Starting Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:05.240 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 podman[60977]: 2024-04-22 06:35:05.067486889 +0000 UTC m=+0.018251313 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:05.240 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 podman[60977]: 2024-04-22 06:35:05.189286208 +0000 UTC m=+0.140050619 container create 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:05.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:05 smithi105 ceph-mon[27916]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:05.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:05 smithi112 ceph-mon[31236]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:05.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:05 smithi177 ceph-mon[31346]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:05.742 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 podman[60977]: 2024-04-22 06:35:05.374967854 +0000 UTC m=+0.325732262 container init 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True) 2024-04-22T06:35:05.743 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 podman[60977]: 2024-04-22 06:35:05.37763893 +0000 UTC m=+0.328403338 container start 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:35:05.743 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 ceph-mds[60996]: -- 172.21.15.177:0/1863509886 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e36cb11a40 con 0x55e36cae5000 2024-04-22T06:35:05.743 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: starting mds.l at 2024-04-22T06:35:05.743 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 bash[60977]: 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 2024-04-22T06:35:05.743 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:35:05 smithi177 systemd[1]: Started Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:06.298 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:06.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- 172.21.15.105:0/2261790682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 msgr2=0x7efef8102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:06.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- 172.21.15.105:0/2261790682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8102450 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efee4009900 tx=0x7efee402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:06.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- 172.21.15.105:0/2261790682 shutdown_connections 2024-04-22T06:35:06.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- 172.21.15.105:0/2261790682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8102990 0x7efef81092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- 172.21.15.105:0/2261790682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8102450 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- 172.21.15.105:0/2261790682 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8100e80 0x7efef81012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- 172.21.15.105:0/2261790682 >> 172.21.15.105:0/2261790682 conn(0x7efef80fc410 msgr2=0x7efef80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- 172.21.15.105:0/2261790682 shutdown_connections 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- 172.21.15.105:0/2261790682 wait complete. 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 Processor -- start 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- start start 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8100e80 0x7efef8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8102990 0x7efef819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efef810db80 con 0x7efef8101fd0 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efef810da00 con 0x7efef8100e80 2024-04-22T06:35:06.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.415+0000 7efefec93640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efef810dd00 con 0x7efef8102990 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45626/0 (socket says 172.21.15.105:45626) 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 -- 172.21.15.105:0/3696027000 learned_addr learned my addr 172.21.15.105:0/3696027000 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efefca2f640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8100e80 0x7efef8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efefd230640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8102990 0x7efef819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 -- 172.21.15.105:0/3696027000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8102990 msgr2=0x7efef819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8102990 0x7efef819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 -- 172.21.15.105:0/3696027000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8100e80 msgr2=0x7efef8198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8100e80 0x7efef8198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 -- 172.21.15.105:0/3696027000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef819fe40 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef7fff640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efee4005ab0 tx=0x7efee40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4043020 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee40054c0 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef81a00d0 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.416+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4003c10 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.417+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef81a06f0 con 0x7efef8101fd0 2024-04-22T06:35:06.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.418+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef8108210 con 0x7efef8101fd0 2024-04-22T06:35:06.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.418+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7efee4046020 con 0x7efef8101fd0 2024-04-22T06:35:06.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.419+0000 7efef5ffb640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 0x7efee80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:06.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.419+0000 7efefca2f640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 0x7efee80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:06.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.419+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7efee40f2230 con 0x7efef8101fd0 2024-04-22T06:35:06.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.419+0000 7efefca2f640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 0x7efee80a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efeec009990 tx=0x7efeec009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:06.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.424+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efee404d020 con 0x7efef8101fd0 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:boot 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: fsmap 1 up:standby 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:06.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:06 smithi112 ceph-mon[31236]: Deploying daemon mds.k on smithi112 2024-04-22T06:35:06.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.640+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efef802d770 con 0x7efef8101fd0 2024-04-22T06:35:06.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.641+0000 7efef5ffb640 1 -- 172.21.15.105:0/3696027000 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7efee40bf0b0 con 0x7efef8101fd0 2024-04-22T06:35:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:06.640 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-22T06:35:06.640 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 2 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 msgr2=0x7efee80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 0x7efee80a60f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efeec009990 tx=0x7efeec009110 comp rx=0 tx=0).stop 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 msgr2=0x7efef8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efee4005ab0 tx=0x7efee40042f0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 shutdown_connections 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efee80a3c30 0x7efee80a60f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efef8102990 0x7efef819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efef8101fd0 0x7efef8198760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 --2- 172.21.15.105:0/3696027000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efef8100e80 0x7efef8198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:06.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 >> 172.21.15.105:0/3696027000 conn(0x7efef80fc410 msgr2=0x7efef81053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:06.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 shutdown_connections 2024-04-22T06:35:06.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:06.643+0000 7efefec93640 1 -- 172.21.15.105:0/3696027000 wait complete. 2024-04-22T06:35:06.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:boot 2024-04-22T06:35:06.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: fsmap 1 up:standby 2024-04-22T06:35:06.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:35:06.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.652 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.653 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.653 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:06.653 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:06.653 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:06.653 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:06 smithi105 ceph-mon[27916]: Deploying daemon mds.k on smithi112 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:boot 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: fsmap 1 up:standby 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:06.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:06 smithi177 ceph-mon[31346]: Deploying daemon mds.k on smithi112 2024-04-22T06:35:07.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:07 smithi105 ceph-mon[27916]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:07.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:07 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3696027000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:07.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:07 smithi112 ceph-mon[31236]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:07.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:07 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3696027000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:07 smithi177 ceph-mon[31346]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:07.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:07 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3696027000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:08.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:35:08 smithi112 systemd[1]: Started Ceph mds.k for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:08.655 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:08.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 -- 172.21.15.105:0/702026728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d01053a0 msgr2=0x7f46d0107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 --2- 172.21.15.105:0/702026728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d01053a0 0x7f46d0107790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009900 tx=0x7f46cc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:08.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 -- 172.21.15.105:0/702026728 shutdown_connections 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 --2- 172.21.15.105:0/702026728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d0107cd0 0x7f46d010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 --2- 172.21.15.105:0/702026728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d01053a0 0x7f46d0107790 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 --2- 172.21.15.105:0/702026728 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0102a70 0x7f46d0104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.755+0000 7f46d79bd640 1 -- 172.21.15.105:0/702026728 >> 172.21.15.105:0/702026728 conn(0x7f46d00fc3e0 msgr2=0x7f46d00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- 172.21.15.105:0/702026728 shutdown_connections 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- 172.21.15.105:0/702026728 wait complete. 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 Processor -- start 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- start start 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 0x7f46d019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0107cd0 0x7f46d01a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010daa0 con 0x7f46d0102a70 2024-04-22T06:35:08.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010d920 con 0x7f46d01053a0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d79bd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010dc20 con 0x7f46d0107cd0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d5759640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d5759640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45642/0 (socket says 172.21.15.105:45642) 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d5759640 1 -- 172.21.15.105:0/380310267 learned_addr learned my addr 172.21.15.105:0/380310267 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d5f5a640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0107cd0 0x7f46d01a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.756+0000 7f46d4f58640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 0x7f46d019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 -- 172.21.15.105:0/380310267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0107cd0 msgr2=0x7f46d01a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0107cd0 0x7f46d01a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 -- 172.21.15.105:0/380310267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 msgr2=0x7f46d019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 0x7f46d019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 -- 172.21.15.105:0/380310267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d01a1fd0 con 0x7f46d0102a70 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d4f58640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 0x7f46d019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d5759640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f46c000d770 tx=0x7f46c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0017070 con 0x7f46d0102a70 2024-04-22T06:35:08.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46c00054d0 con 0x7f46d0102a70 2024-04-22T06:35:08.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d01a22c0 con 0x7f46d0102a70 2024-04-22T06:35:08.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0004c30 con 0x7f46d0102a70 2024-04-22T06:35:08.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.757+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d01a2790 con 0x7f46d0102a70 2024-04-22T06:35:08.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.758+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d01097c0 con 0x7f46d0102a70 2024-04-22T06:35:08.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.759+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f46c0004e10 con 0x7f46d0102a70 2024-04-22T06:35:08.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.760+0000 7f46c67fc640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 0x7f46bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:08.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.760+0000 7f46d4f58640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 0x7f46bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:08.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.760+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f46c00cfa20 con 0x7f46d0102a70 2024-04-22T06:35:08.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.760+0000 7f46d4f58640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 0x7f46bc0a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46cc006010 tx=0x7f46cc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:08.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.765+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46c009c8a0 con 0x7f46d0102a70 2024-04-22T06:35:08.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.993+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46d002d5c0 con 0x7f46d0102a70 2024-04-22T06:35:08.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.994+0000 7f46c67fc640 1 -- 172.21.15.105:0/380310267 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f46c009c1a0 con 0x7f46d0102a70 2024-04-22T06:35:08.993 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:08.993 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:35:08.993 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 3 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 msgr2=0x7f46bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 0x7f46bc0a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46cc006010 tx=0x7f46cc02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 msgr2=0x7f46d019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f46c000d770 tx=0x7f46c000dc30 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 shutdown_connections 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f46bc0a3bb0 0x7f46bc0a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46d0107cd0 0x7f46d01a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46d01053a0 0x7f46d019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 --2- 172.21.15.105:0/380310267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46d0102a70 0x7f46d019a3b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 >> 172.21.15.105:0/380310267 conn(0x7f46d00fc3e0 msgr2=0x7f46d01021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:08.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 shutdown_connections 2024-04-22T06:35:08.996 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:08.996+0000 7f46d79bd640 1 -- 172.21.15.105:0/380310267 wait complete. 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:09.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: Deploying daemon mds.j on smithi105 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:boot 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: fsmap 2 up:standby 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:09.590 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:09 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/380310267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:09.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: Deploying daemon mds.j on smithi105 2024-04-22T06:35:09.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:boot 2024-04-22T06:35:09.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: fsmap 2 up:standby 2024-04-22T06:35:09.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:35:09.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:09.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:09 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/380310267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: Deploying daemon mds.j on smithi105 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:boot 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: fsmap 2 up:standby 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:35:09.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:09.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:09 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/380310267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:10.668 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 systemd[1]: Starting Ceph mds.j for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:10.977 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 podman[68864]: 2024-04-22 06:35:10.609732943 +0000 UTC m=+0.017731288 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 podman[68864]: 2024-04-22 06:35:10.709845344 +0000 UTC m=+0.117843738 container create b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 podman[68864]: 2024-04-22 06:35:10.904183001 +0000 UTC m=+0.312181339 container init b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 podman[68864]: 2024-04-22 06:35:10.906628628 +0000 UTC m=+0.314626967 container start b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 ceph-mds[68883]: -- 172.21.15.105:0/357404194 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ac03c9a40 con 0x564ac039cc00 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j[68879]: starting mds.j at 2024-04-22T06:35:10.978 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 bash[68864]: b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 2024-04-22T06:35:11.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- 172.21.15.105:0/4283914676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90101fc0 msgr2=0x7f9a90102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4283914676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90101fc0 0x7f9a90102440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009900 tx=0x7f9a8402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- 172.21.15.105:0/4283914676 shutdown_connections 2024-04-22T06:35:11.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4283914676 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90102980 0x7f9a901092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4283914676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90101fc0 0x7f9a90102440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4283914676 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90100e70 0x7f9a90101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- 172.21.15.105:0/4283914676 >> 172.21.15.105:0/4283914676 conn(0x7f9a900fc3e0 msgr2=0x7f9a900fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- 172.21.15.105:0/4283914676 shutdown_connections 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- 172.21.15.105:0/4283914676 wait complete. 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 Processor -- start 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.149+0000 7f9a963f4640 1 -- start start 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90100e70 0x7f9a90104f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90101fc0 0x7f9a901035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9010c9d0 con 0x7f9a90102980 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9010c850 con 0x7f9a90101fc0 2024-04-22T06:35:11.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9010cb50 con 0x7f9a90100e70 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45668/0 (socket says 172.21.15.105:45668) 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 -- 172.21.15.105:0/4279054635 learned_addr learned my addr 172.21.15.105:0/4279054635 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a8f7fe640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90101fc0 0x7f9a901035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a8ffff640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90100e70 0x7f9a90104f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 -- 172.21.15.105:0/4279054635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90100e70 msgr2=0x7f9a90104f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90100e70 0x7f9a90104f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 -- 172.21.15.105:0/4279054635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90101fc0 msgr2=0x7f9a901035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90101fc0 0x7f9a901035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 -- 172.21.15.105:0/4279054635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a901043a0 con 0x7f9a90102980 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a94991640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cad0 tx=0x7f9a8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800ae20 con 0x7f9a90102980 2024-04-22T06:35:11.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a901a22d0 con 0x7f9a90102980 2024-04-22T06:35:11.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88005020 con 0x7f9a90102980 2024-04-22T06:35:11.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800f6b0 con 0x7f9a90102980 2024-04-22T06:35:11.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.150+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a901a2840 con 0x7f9a90102980 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.151+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a901a2b60 con 0x7f9a90102980 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.153+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9a880051d0 con 0x7f9a90102980 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.153+0000 7f9a8d7fa640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 0x7f9a780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.154+0000 7f9a8ffff640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 0x7f9a780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.154+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f9a880d02b0 con 0x7f9a90102980 2024-04-22T06:35:11.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.154+0000 7f9a8ffff640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 0x7f9a780a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000bbe0 tx=0x7f9a8000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:11.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.157+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a88014030 con 0x7f9a90102980 2024-04-22T06:35:11.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.375+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a901a3170 con 0x7f9a90102980 2024-04-22T06:35:11.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.375+0000 7f9a8d7fa640 1 -- 172.21.15.105:0/4279054635 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f9a8809cae0 con 0x7f9a90102980 2024-04-22T06:35:11.374 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:11.374 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:35:11.374 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 3 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 msgr2=0x7f9a780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 0x7f9a780a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000bbe0 tx=0x7f9a8000b390 comp rx=0 tx=0).stop 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 msgr2=0x7f9a90103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800cad0 tx=0x7f9a8800cf90 comp rx=0 tx=0).stop 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 shutdown_connections 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9a780a3bb0 0x7f9a780a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a90102980 0x7f9a90103b10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a90101fc0 0x7f9a901035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 --2- 172.21.15.105:0/4279054635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a90100e70 0x7f9a90104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:11.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 >> 172.21.15.105:0/4279054635 conn(0x7f9a900fc3e0 msgr2=0x7f9a900690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:11.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 shutdown_connections 2024-04-22T06:35:11.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:11.377+0000 7f9a963f4640 1 -- 172.21.15.105:0/4279054635 wait complete. 2024-04-22T06:35:11.386 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:35:10 smithi105 systemd[1]: Started Ceph mds.j for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:11.922 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:11 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4279054635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:11.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:11 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4279054635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:11 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4279054635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:12.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:12 smithi112 ceph-mon[31236]: Deploying daemon mds.d on smithi105 2024-04-22T06:35:12.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:12 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6834/3495232000,v1:172.21.15.105:6835/3495232000] up:boot 2024-04-22T06:35:12.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:12 smithi112 ceph-mon[31236]: fsmap 3 up:standby 2024-04-22T06:35:12.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:12 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:35:12.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:12 smithi177 ceph-mon[31346]: Deploying daemon mds.d on smithi105 2024-04-22T06:35:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:12 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6834/3495232000,v1:172.21.15.105:6835/3495232000] up:boot 2024-04-22T06:35:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:12 smithi177 ceph-mon[31346]: fsmap 3 up:standby 2024-04-22T06:35:12.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:12 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:35:12.993 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:12 smithi105 ceph-mon[27916]: Deploying daemon mds.d on smithi105 2024-04-22T06:35:12.993 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:12 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6834/3495232000,v1:172.21.15.105:6835/3495232000] up:boot 2024-04-22T06:35:12.993 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:12 smithi105 ceph-mon[27916]: fsmap 3 up:standby 2024-04-22T06:35:12.994 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:12 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:35:13.389 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:13.409 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 systemd[1]: Starting Ceph mds.d for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 -- 172.21.15.105:0/3227239358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103170 msgr2=0x7f829c1035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 --2- 172.21.15.105:0/3227239358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103170 0x7f829c1035f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f828c009900 tx=0x7f828c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 -- 172.21.15.105:0/3227239358 shutdown_connections 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 --2- 172.21.15.105:0/3227239358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103b30 0x7f829c10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 --2- 172.21.15.105:0/3227239358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103170 0x7f829c1035f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 --2- 172.21.15.105:0/3227239358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 0x7f829c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 -- 172.21.15.105:0/3227239358 >> 172.21.15.105:0/3227239358 conn(0x7f829c0fd530 msgr2=0x7f829c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:13.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 -- 172.21.15.105:0/3227239358 shutdown_connections 2024-04-22T06:35:13.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 -- 172.21.15.105:0/3227239358 wait complete. 2024-04-22T06:35:13.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.493+0000 7f82a125c640 1 Processor -- start 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 -- start start 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 0x7f829c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103170 0x7f829c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10db00 con 0x7f829c103b30 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10d980 con 0x7f829c101f80 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f82a125c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f829c10dc80 con 0x7f829c103170 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45690/0 (socket says 172.21.15.105:45690) 2024-04-22T06:35:13.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 -- 172.21.15.105:0/1364223817 learned_addr learned my addr 172.21.15.105:0/1364223817 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829a59c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103170 0x7f829c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829ad9d640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 0x7f829c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 -- 172.21.15.105:0/1364223817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103170 msgr2=0x7f829c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103170 0x7f829c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 -- 172.21.15.105:0/1364223817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 msgr2=0x7f829c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 0x7f829c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 -- 172.21.15.105:0/1364223817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829c19fe40 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.494+0000 7f829b59e640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f829400b970 tx=0x7f829400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.495+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294019070 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.495+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829c1a0130 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.495+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8294014440 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.495+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82940075d0 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.495+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829c1a0720 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.496+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829c1093c0 con 0x7f829c103b30 2024-04-22T06:35:13.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.497+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f829401c070 con 0x7f829c103b30 2024-04-22T06:35:13.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.497+0000 7f8283fff640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 0x7f82840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:13.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.497+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f82940cf4f0 con 0x7f829c103b30 2024-04-22T06:35:13.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.497+0000 7f829ad9d640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 0x7f82840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:13.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.498+0000 7f829ad9d640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 0x7f82840a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8290005fd0 tx=0x7f8290005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:13.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.502+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f829409bd20 con 0x7f829c103b30 2024-04-22T06:35:13.668 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 podman[69212]: 2024-04-22 06:35:13.314138443 +0000 UTC m=+0.018008663 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:13.668 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 podman[69212]: 2024-04-22 06:35:13.422763735 +0000 UTC m=+0.126633945 container create 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:35:13.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.726+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f829c062b50 con 0x7f829c103b30 2024-04-22T06:35:13.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.726+0000 7f8283fff640 1 -- 172.21.15.105:0/1364223817 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f82940a0020 con 0x7f829c103b30 2024-04-22T06:35:13.725 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:13.725 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:35:13.725 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 4 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 msgr2=0x7f82840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 0x7f82840a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8290005fd0 tx=0x7f8290005850 comp rx=0 tx=0).stop 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 msgr2=0x7f829c19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f829400b970 tx=0x7f829400be30 comp rx=0 tx=0).stop 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 shutdown_connections 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f82840a3bb0 0x7f82840a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f829c103b30 0x7f829c19f7d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f829c103170 0x7f829c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 --2- 172.21.15.105:0/1364223817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f829c101f80 0x7f829c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 >> 172.21.15.105:0/1364223817 conn(0x7f829c0fd530 msgr2=0x7f829c106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 shutdown_connections 2024-04-22T06:35:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:13.728+0000 7f82a125c640 1 -- 172.21.15.105:0/1364223817 wait complete. 2024-04-22T06:35:13.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:13 smithi112 ceph-mon[31236]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:13.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:13 smithi177 ceph-mon[31346]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:14.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:13 smithi105 ceph-mon[27916]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:14.168 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 podman[69212]: 2024-04-22 06:35:13.792615259 +0000 UTC m=+0.496485468 container init 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:35:14.168 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 podman[69212]: 2024-04-22 06:35:13.795336398 +0000 UTC m=+0.499206609 container start 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:35:14.168 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 ceph-mds[69272]: -- 172.21.15.105:0/1399616303 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5587878eba40 con 0x5587878be800 2024-04-22T06:35:14.168 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d[69266]: starting mds.d at 2024-04-22T06:35:14.169 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 bash[69212]: 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 2024-04-22T06:35:14.169 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:35:13 smithi105 systemd[1]: Started Ceph mds.d for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:14.856 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1364223817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:14.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:14 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1364223817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:14 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1364223817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:15.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:15.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:15.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:15.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:15.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:14 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:15.739 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/4265849499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4103180 msgr2=0x7f2ec4103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/4265849499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4103180 0x7f2ec4103600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009900 tx=0x7f2eac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/4265849499 shutdown_connections 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/4265849499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 0x7f2ec410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/4265849499 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4103180 0x7f2ec4103600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/4265849499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4101f90 0x7f2ec41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/4265849499 >> 172.21.15.105:0/4265849499 conn(0x7f2ec40fd500 msgr2=0x7f2ec40ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:15.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/4265849499 shutdown_connections 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.851+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/4265849499 wait complete. 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 Processor -- start 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 -- start start 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4103180 0x7f2ec4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 0x7f2ec4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410dad0 con 0x7f2ec4101f90 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410d950 con 0x7f2ec4103180 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec9c5c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec410dc50 con 0x7f2ec4103b40 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45428/0 (socket says 172.21.15.105:45428) 2024-04-22T06:35:15.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 -- 172.21.15.105:0/2324808883 learned_addr learned my addr 172.21.15.105:0/2324808883 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ebbfff640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4103180 0x7f2ec4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec3fff640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 0x7f2ec4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 -- 172.21.15.105:0/2324808883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 msgr2=0x7f2ec4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 0x7f2ec4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 -- 172.21.15.105:0/2324808883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4103180 msgr2=0x7f2ec4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4103180 0x7f2ec4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 -- 172.21.15.105:0/2324808883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec4075110 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.852+0000 7f2ec37fe640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400d770 tx=0x7f2eb400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.853+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4017070 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.853+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec41a2200 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.853+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eb4009c30 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.853+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003cc0 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.853+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec41a2770 con 0x7f2ec4101f90 2024-04-22T06:35:15.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.854+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec41092b0 con 0x7f2ec4101f90 2024-04-22T06:35:15.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.855+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2eb4003e60 con 0x7f2ec4101f90 2024-04-22T06:35:15.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.855+0000 7f2ec17fa640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 0x7f2eb00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:15.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.855+0000 7f2ebbfff640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 0x7f2eb00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:15.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.855+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f2eb40cf8f0 con 0x7f2ec4101f90 2024-04-22T06:35:15.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.856+0000 7f2ebbfff640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 0x7f2eb00a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009230 tx=0x7f2eac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:15.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:15.860+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2eb409c770 con 0x7f2ec4101f90 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:15 smithi112 ceph-mon[31236]: Deploying daemon mds.e on smithi112 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:15 smithi112 ceph-mon[31236]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:15 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6836/2718881173,v1:172.21.15.105:6837/2718881173] up:boot 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:15 smithi112 ceph-mon[31236]: fsmap 4 up:standby 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:15 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:35:15.959 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:15 smithi112 systemd[1]: Starting Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:15 smithi177 ceph-mon[31346]: Deploying daemon mds.e on smithi112 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:15 smithi177 ceph-mon[31346]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:15 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6836/2718881173,v1:172.21.15.105:6837/2718881173] up:boot 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:15 smithi177 ceph-mon[31346]: fsmap 4 up:standby 2024-04-22T06:35:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:15 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:35:16.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.073+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ec4062b50 con 0x7f2ec4101f90 2024-04-22T06:35:16.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.073+0000 7f2ec17fa640 1 -- 172.21.15.105:0/2324808883 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f2eb4009de0 con 0x7f2ec4101f90 2024-04-22T06:35:16.073 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:16.074 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-22T06:35:16.074 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 5 2024-04-22T06:35:16.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 msgr2=0x7f2eb00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:16.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 0x7f2eb00a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009230 tx=0x7f2eac005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:16.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 msgr2=0x7f2ec4075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400d770 tx=0x7f2eb400dc30 comp rx=0 tx=0).stop 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 shutdown_connections 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2eb00a3bb0 0x7f2eb00a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ec4103b40 0x7f2ec4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ec4103180 0x7f2ec4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 --2- 172.21.15.105:0/2324808883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ec4101f90 0x7f2ec4075d00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.075+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 >> 172.21.15.105:0/2324808883 conn(0x7f2ec40fd500 msgr2=0x7f2ec4106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.076+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 shutdown_connections 2024-04-22T06:35:16.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:16.076+0000 7f2ec9c5c640 1 -- 172.21.15.105:0/2324808883 wait complete. 2024-04-22T06:35:16.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:15 smithi105 ceph-mon[27916]: Deploying daemon mds.e on smithi112 2024-04-22T06:35:16.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:15 smithi105 ceph-mon[27916]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:16.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:15 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6836/2718881173,v1:172.21.15.105:6837/2718881173] up:boot 2024-04-22T06:35:16.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:15 smithi105 ceph-mon[27916]: fsmap 4 up:standby 2024-04-22T06:35:16.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:15 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:35:16.373 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 podman[60404]: 2024-04-22 06:35:16.016976607 +0000 UTC m=+0.018130746 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:16.373 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 podman[60404]: 2024-04-22 06:35:16.147808374 +0000 UTC m=+0.148962533 container create bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:35:16.373 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 podman[60404]: 2024-04-22 06:35:16.350211915 +0000 UTC m=+0.351366030 container init bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T06:35:16.373 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 podman[60404]: 2024-04-22 06:35:16.352851145 +0000 UTC m=+0.354005279 container start bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:35:16.685 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mds[60423]: -- 172.21.15.112:0/3123928204 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f31701a40 con 0x556f316d5000 2024-04-22T06:35:16.685 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: starting mds.e at 2024-04-22T06:35:16.685 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 bash[60404]: bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a 2024-04-22T06:35:16.685 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:35:16 smithi112 systemd[1]: Started Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2324808883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:16.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:16 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2324808883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:16 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2324808883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:16 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:17.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:17 smithi112 ceph-mon[31236]: Deploying daemon mds.f on smithi177 2024-04-22T06:35:17.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:17 smithi112 ceph-mon[31236]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:17 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:boot 2024-04-22T06:35:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:17 smithi112 ceph-mon[31236]: fsmap 5 up:standby 2024-04-22T06:35:17.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:17 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:35:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:17 smithi177 ceph-mon[31346]: Deploying daemon mds.f on smithi177 2024-04-22T06:35:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:17 smithi177 ceph-mon[31346]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:17 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:boot 2024-04-22T06:35:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:17 smithi177 ceph-mon[31346]: fsmap 5 up:standby 2024-04-22T06:35:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:17 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:35:18.087 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:18.112 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:17 smithi105 ceph-mon[27916]: Deploying daemon mds.f on smithi177 2024-04-22T06:35:18.112 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:17 smithi105 ceph-mon[27916]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:18.112 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:17 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:boot 2024-04-22T06:35:18.112 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:17 smithi105 ceph-mon[27916]: fsmap 5 up:standby 2024-04-22T06:35:18.112 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:17 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:35:18.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 -- 172.21.15.105:0/428415461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 msgr2=0x7f28fc1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 --2- 172.21.15.105:0/428415461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc1023c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009900 tx=0x7f28f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 -- 172.21.15.105:0/428415461 shutdown_connections 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 --2- 172.21.15.105:0/428415461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103b50 0x7f28fc10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 --2- 172.21.15.105:0/428415461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103190 0x7f28fc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 --2- 172.21.15.105:0/428415461 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc1023c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.197+0000 7f29046e0640 1 -- 172.21.15.105:0/428415461 >> 172.21.15.105:0/428415461 conn(0x7f28fc0fd530 msgr2=0x7f28fc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- 172.21.15.105:0/428415461 shutdown_connections 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- 172.21.15.105:0/428415461 wait complete. 2024-04-22T06:35:18.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 Processor -- start 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- start start 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 0x7f28fc074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103b50 0x7f28fc074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10dae0 con 0x7f28fc101fa0 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10d960 con 0x7f28fc103190 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f29046e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc10dc60 con 0x7f28fc103b50 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f290247c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f290247c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45450/0 (socket says 172.21.15.105:45450) 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f290247c640 1 -- 172.21.15.105:0/1378235245 learned_addr learned my addr 172.21.15.105:0/1378235245 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.198+0000 7f2902c7d640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103b50 0x7f28fc074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f2901c7b640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 0x7f28fc074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 -- 172.21.15.105:0/1378235245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103b50 msgr2=0x7f28fc074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103b50 0x7f28fc074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 -- 172.21.15.105:0/1378235245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 msgr2=0x7f28fc074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 0x7f28fc074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 -- 172.21.15.105:0/1378235245 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc075110 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f2901c7b640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 0x7f28fc074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f290247c640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8043020 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28f8004440 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc1a22e0 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8003c90 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.199+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1a2880 con 0x7f28fc101fa0 2024-04-22T06:35:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.200+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc109340 con 0x7f28fc101fa0 2024-04-22T06:35:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.201+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f28f8046020 con 0x7f28fc101fa0 2024-04-22T06:35:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.202+0000 7f28f37fe640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 0x7f28e80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.202+0000 7f2901c7b640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 0x7f28e80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.202+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f28f80f25a0 con 0x7f28fc101fa0 2024-04-22T06:35:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.202+0000 7f2901c7b640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 0x7f28e80a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f28fc075a70 tx=0x7f28ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:18.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.206+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28f804d020 con 0x7f28fc101fa0 2024-04-22T06:35:18.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.423+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28fc02d5c0 con 0x7f28fc101fa0 2024-04-22T06:35:18.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.423+0000 7f28f37fe640 1 -- 172.21.15.105:0/1378235245 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f28f80bf420 con 0x7f28fc101fa0 2024-04-22T06:35:18.423 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:18.423 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-22T06:35:18.423 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 6 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.425+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 msgr2=0x7f28e80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.425+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 0x7f28e80a60f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f28fc075a70 tx=0x7f28ec005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.425+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 msgr2=0x7f28fc075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.425+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28f8000c00 tx=0x7f28f8004040 comp rx=0 tx=0).stop 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 shutdown_connections 2024-04-22T06:35:18.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f28e80a3c30 0x7f28e80a60f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f28fc103b50 0x7f28fc074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f28fc103190 0x7f28fc074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 --2- 172.21.15.105:0/1378235245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f28fc101fa0 0x7f28fc075d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 >> 172.21.15.105:0/1378235245 conn(0x7f28fc0fd530 msgr2=0x7f28fc106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 shutdown_connections 2024-04-22T06:35:18.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:18.426+0000 7f29046e0640 1 -- 172.21.15.105:0/1378235245 wait complete. 2024-04-22T06:35:18.675 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 systemd[1]: Starting Ceph mds.f for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:18.958 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:18 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1378235245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:18.958 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 podman[61283]: 2024-04-22 06:35:18.57940089 +0000 UTC m=+0.018056855 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 podman[61283]: 2024-04-22 06:35:18.691620651 +0000 UTC m=+0.130276621 container create b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 podman[61283]: 2024-04-22 06:35:18.893945542 +0000 UTC m=+0.332601477 container init b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 podman[61283]: 2024-04-22 06:35:18.896499107 +0000 UTC m=+0.335155038 container start b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 ceph-mds[61302]: -- 172.21.15.177:0/3770899715 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55afca005a40 con 0x55afc9fd9000 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: starting mds.f at 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 bash[61283]: b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 2024-04-22T06:35:18.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:18 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1378235245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:19.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:18 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1378235245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:19.239 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:35:18 smithi177 systemd[1]: Started Ceph mds.f for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:19.937 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:19.938 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:19 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:19.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:19.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:19 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:20.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:20.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:20.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:20.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:20.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:20.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:20.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:19 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:20.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:20.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.537+0000 7f88faa3e640 1 -- 172.21.15.105:0/3829498596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4100e70 msgr2=0x7f88f4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.537+0000 7f88faa3e640 1 --2- 172.21.15.105:0/3829498596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4100e70 0x7f88f4101290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009900 tx=0x7f88e402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- 172.21.15.105:0/3829498596 shutdown_connections 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- 172.21.15.105:0/3829498596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4102980 0x7f88f41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- 172.21.15.105:0/3829498596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4101fc0 0x7f88f4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- 172.21.15.105:0/3829498596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4100e70 0x7f88f4101290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- 172.21.15.105:0/3829498596 >> 172.21.15.105:0/3829498596 conn(0x7f88f40fc3e0 msgr2=0x7f88f40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- 172.21.15.105:0/3829498596 shutdown_connections 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- 172.21.15.105:0/3829498596 wait complete. 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 Processor -- start 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- start start 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4100e70 0x7f88f4069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:20.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4102980 0x7f88f419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410c9c0 con 0x7f88f4101fc0 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410c840 con 0x7f88f4102980 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.538+0000 7f88faa3e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f88f410cb40 con 0x7f88f4100e70 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45458/0 (socket says 172.21.15.105:45458) 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 -- 172.21.15.105:0/304787096 learned_addr learned my addr 172.21.15.105:0/304787096 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f3fff640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4100e70 0x7f88f4069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f8fdb640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4102980 0x7f88f419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 -- 172.21.15.105:0/304787096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4100e70 msgr2=0x7f88f4069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4100e70 0x7f88f4069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 -- 172.21.15.105:0/304787096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4102980 msgr2=0x7f88f419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4102980 0x7f88f419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 -- 172.21.15.105:0/304787096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f419fe10 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f37fe640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f88e800d770 tx=0x7f88e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8017070 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f41a00a0 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88e8009c30 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8003cc0 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.539+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f41a0640 con 0x7f88f4101fc0 2024-04-22T06:35:20.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.540+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f4108220 con 0x7f88f4101fc0 2024-04-22T06:35:20.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.541+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f88e8003e60 con 0x7f88f4101fc0 2024-04-22T06:35:20.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.542+0000 7f88f17fa640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 0x7f88dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:20.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.542+0000 7f88f3fff640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 0x7f88dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:20.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.542+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f88e80cf840 con 0x7f88f4101fc0 2024-04-22T06:35:20.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.542+0000 7f88f3fff640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 0x7f88dc0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009230 tx=0x7f88e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:20.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.546+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88e8009de0 con 0x7f88f4101fc0 2024-04-22T06:35:20.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.762+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88f402d7c0 con 0x7f88f4101fc0 2024-04-22T06:35:20.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.763+0000 7f88f17fa640 1 -- 172.21.15.105:0/304787096 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f88e809c070 con 0x7f88f4101fc0 2024-04-22T06:35:20.763 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:20.763 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-22T06:35:20.763 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 7 2024-04-22T06:35:20.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 msgr2=0x7f88dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 0x7f88dc0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009230 tx=0x7f88e4005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 msgr2=0x7f88f419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f88e800d770 tx=0x7f88e800dc30 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 shutdown_connections 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f88dc0a3bb0 0x7f88dc0a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88f4102980 0x7f88f419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88f4101fc0 0x7f88f419cee0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 --2- 172.21.15.105:0/304787096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f88f4100e70 0x7f88f4069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 >> 172.21.15.105:0/304787096 conn(0x7f88f40fc3e0 msgr2=0x7f88f41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 shutdown_connections 2024-04-22T06:35:20.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:20.765+0000 7f88faa3e640 1 -- 172.21.15.105:0/304787096 wait complete. 2024-04-22T06:35:20.859 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:20 smithi177 ceph-mon[31346]: Deploying daemon mds.c on smithi177 2024-04-22T06:35:20.859 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:20 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6834/3449982583,v1:172.21.15.177:6835/3449982583] up:boot 2024-04-22T06:35:20.859 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:20 smithi177 ceph-mon[31346]: fsmap 6 up:standby 2024-04-22T06:35:20.860 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:20 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:35:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:20 smithi112 ceph-mon[31236]: Deploying daemon mds.c on smithi177 2024-04-22T06:35:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:20 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6834/3449982583,v1:172.21.15.177:6835/3449982583] up:boot 2024-04-22T06:35:20.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:20 smithi112 ceph-mon[31236]: fsmap 6 up:standby 2024-04-22T06:35:20.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:20 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:35:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:20 smithi105 ceph-mon[27916]: Deploying daemon mds.c on smithi177 2024-04-22T06:35:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:20 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6834/3449982583,v1:172.21.15.177:6835/3449982583] up:boot 2024-04-22T06:35:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:20 smithi105 ceph-mon[27916]: fsmap 6 up:standby 2024-04-22T06:35:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:20 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:35:21.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 systemd[1]: Starting Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:21.512 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 podman[61584]: 2024-04-22 06:35:21.162689788 +0000 UTC m=+0.017828146 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:21.513 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 podman[61584]: 2024-04-22 06:35:21.269818395 +0000 UTC m=+0.124956788 container create 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-04-22T06:35:21.513 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 podman[61584]: 2024-04-22 06:35:21.488823142 +0000 UTC m=+0.343961490 container init 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:35:21.513 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 podman[61584]: 2024-04-22 06:35:21.49142628 +0000 UTC m=+0.346564638 container start 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-22T06:35:21.513 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mds[61603]: -- 172.21.15.177:0/2793216632 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9ea0d3a40 con 0x55d9ea0a6800 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: starting mds.c at 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 bash[61584]: 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:35:21 smithi177 systemd[1]: Started Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/304787096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:21.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:21.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:21.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:21 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/304787096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:21 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/304787096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:21 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:22.776 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 -- 172.21.15.105:0/1359284509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f8103150 msgr2=0x7fa1f81055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 --2- 172.21.15.105:0/1359284509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f8103150 0x7fa1f81055c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009900 tx=0x7fa1ec02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 -- 172.21.15.105:0/1359284509 shutdown_connections 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 --2- 172.21.15.105:0/1359284509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8105b00 0x7fa1f8105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 --2- 172.21.15.105:0/1359284509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f8103150 0x7fa1f81055c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 --2- 172.21.15.105:0/1359284509 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f81007e0 0x7fa1f8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 -- 172.21.15.105:0/1359284509 >> 172.21.15.105:0/1359284509 conn(0x7fa1f80fc3e0 msgr2=0x7fa1f80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 -- 172.21.15.105:0/1359284509 shutdown_connections 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.886+0000 7fa20017e640 1 -- 172.21.15.105:0/1359284509 wait complete. 2024-04-22T06:35:22.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 Processor -- start 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 -- start start 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8103150 0x7fa1f819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f8105b00 0x7fa1f81a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f81140c0 con 0x7fa1f81007e0 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f8113f40 con 0x7fa1f8103150 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa20017e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f8114240 con 0x7fa1f8105b00 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45474/0 (socket says 172.21.15.105:45474) 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 -- 172.21.15.105:0/3265213338 learned_addr learned my addr 172.21.15.105:0/3265213338 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fd719640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8103150 0x7fa1f819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 -- 172.21.15.105:0/3265213338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f8105b00 msgr2=0x7fa1f81a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fe71b640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f8105b00 0x7fa1f81a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f8105b00 0x7fa1f81a5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:22.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 -- 172.21.15.105:0/3265213338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8103150 msgr2=0x7fa1f819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8103150 0x7fa1f819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.887+0000 7fa1fdf1a640 1 -- 172.21.15.105:0/3265213338 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f81a6370 con 0x7fa1f81007e0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa1fdf1a640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000ea80 tx=0x7fa1f000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000ce40 con 0x7fa1f81007e0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f81a6660 con 0x7fa1f81007e0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f0005020 con 0x7fa1f81007e0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00116d0 con 0x7fa1f81007e0 2024-04-22T06:35:22.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.888+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f81a6b80 con 0x7fa1f81007e0 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.889+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f80fded0 con 0x7fa1f81007e0 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.890+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa1f00054d0 con 0x7fa1f81007e0 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.890+0000 7fa1f6ffd640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 0x7fa1e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.891+0000 7fa1fd719640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 0x7fa1e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.891+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fa1f00d02d0 con 0x7fa1f81007e0 2024-04-22T06:35:22.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.891+0000 7fa1fd719640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 0x7fa1e00a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009230 tx=0x7fa1ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:22.893 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:22.895+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1f0016030 con 0x7fa1f81007e0 2024-04-22T06:35:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:22 smithi112 ceph-mon[31236]: Deploying daemon mds.h on smithi112 2024-04-22T06:35:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:22 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:boot 2024-04-22T06:35:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:22 smithi112 ceph-mon[31236]: fsmap 7 up:standby 2024-04-22T06:35:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:22 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:35:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:22 smithi177 ceph-mon[31346]: Deploying daemon mds.h on smithi112 2024-04-22T06:35:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:22 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:boot 2024-04-22T06:35:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:22 smithi177 ceph-mon[31346]: fsmap 7 up:standby 2024-04-22T06:35:22.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:22 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:35:23.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.111+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1f8062b50 con 0x7fa1f81007e0 2024-04-22T06:35:23.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.111+0000 7fa1f6ffd640 1 -- 172.21.15.105:0/3265213338 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fa1f009d0b0 con 0x7fa1f81007e0 2024-04-22T06:35:23.110 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:23.110 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-22T06:35:23.110 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 8 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 msgr2=0x7fa1e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 0x7fa1e00a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009230 tx=0x7fa1ec005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 msgr2=0x7fa1f819e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000ea80 tx=0x7fa1f000ef40 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 shutdown_connections 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1e00a3bb0 0x7fa1e00a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1f8105b00 0x7fa1f81a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1f8103150 0x7fa1f819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 --2- 172.21.15.105:0/3265213338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1f81007e0 0x7fa1f819e750 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 >> 172.21.15.105:0/3265213338 conn(0x7fa1f80fc3e0 msgr2=0x7fa1f8101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 shutdown_connections 2024-04-22T06:35:23.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:23.113+0000 7fa20017e640 1 -- 172.21.15.105:0/3265213338 wait complete. 2024-04-22T06:35:23.121 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:22 smithi105 ceph-mon[27916]: Deploying daemon mds.h on smithi112 2024-04-22T06:35:23.121 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:22 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:boot 2024-04-22T06:35:23.121 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:22 smithi105 ceph-mon[27916]: fsmap 7 up:standby 2024-04-22T06:35:23.122 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:22 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:35:23.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:23 smithi112 ceph-mon[31236]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:23.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:23 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3265213338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:23.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:23 smithi112 systemd[1]: Starting Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:23 smithi177 ceph-mon[31346]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:23 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3265213338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:23 smithi105 ceph-mon[27916]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:23 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3265213338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:24.276 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 podman[60716]: 2024-04-22 06:35:23.905736579 +0000 UTC m=+0.017982998 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 podman[60716]: 2024-04-22 06:35:24.009493575 +0000 UTC m=+0.121739979 container create ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True) 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 podman[60716]: 2024-04-22 06:35:24.220214864 +0000 UTC m=+0.332461269 container init ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 podman[60716]: 2024-04-22 06:35:24.222816006 +0000 UTC m=+0.335062410 container start ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 ceph-mds[60735]: -- 172.21.15.112:0/772913580 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c43a58da40 con 0x55c43a561000 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: starting mds.h at 2024-04-22T06:35:24.277 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 bash[60716]: ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b 2024-04-22T06:35:24.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:35:24 smithi112 systemd[1]: Started Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:25.125 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:25.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- 172.21.15.105:0/4067807396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 msgr2=0x7fa768102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 --2- 172.21.15.105:0/4067807396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768102440 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa764009900 tx=0x7fa76402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:25.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- 172.21.15.105:0/4067807396 shutdown_connections 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 --2- 172.21.15.105:0/4067807396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 0x7fa7681092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 --2- 172.21.15.105:0/4067807396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768102440 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 --2- 172.21.15.105:0/4067807396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa768101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- 172.21.15.105:0/4067807396 >> 172.21.15.105:0/4067807396 conn(0x7fa7680fc3e0 msgr2=0x7fa7680fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- 172.21.15.105:0/4067807396 shutdown_connections 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- 172.21.15.105:0/4067807396 wait complete. 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 Processor -- start 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.236+0000 7fa770104640 1 -- start start 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa7681981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 0x7fa76819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa76810c9d0 con 0x7fa768101fc0 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa76810c850 con 0x7fa768100e70 2024-04-22T06:35:25.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa76810cb50 con 0x7fa768102980 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38970/0 (socket says 172.21.15.105:38970) 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 -- 172.21.15.105:0/3849754986 learned_addr learned my addr 172.21.15.105:0/3849754986 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76e6a1640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 0x7fa76819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76dea0640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa7681981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 -- 172.21.15.105:0/3849754986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 msgr2=0x7fa76819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 0x7fa76819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 -- 172.21.15.105:0/3849754986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 msgr2=0x7fa7681981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa7681981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 -- 172.21.15.105:0/3849754986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76819fdf0 con 0x7fa768101fc0 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76dea0640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa7681981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa76d69f640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa764000c00 tx=0x7fa764004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764043020 con 0x7fa768101fc0 2024-04-22T06:35:25.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7681a0080 con 0x7fa768101fc0 2024-04-22T06:35:25.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa764004440 con 0x7fa768101fc0 2024-04-22T06:35:25.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764003c90 con 0x7fa768101fc0 2024-04-22T06:35:25.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.237+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7681a0620 con 0x7fa768101fc0 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.238+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7681a0940 con 0x7fa768101fc0 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.239+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa764046020 con 0x7fa768101fc0 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.240+0000 7fa75effd640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 0x7fa7540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.240+0000 7fa76dea0640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 0x7fa7540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.240+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fa7640f21b0 con 0x7fa768101fc0 2024-04-22T06:35:25.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.240+0000 7fa76dea0640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 0x7fa7540a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa7580058c0 tx=0x7fa758005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:25.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.244+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa76404d020 con 0x7fa768101fc0 2024-04-22T06:35:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: Deploying daemon mds.i on smithi177 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:boot 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: fsmap 8 up:standby 2024-04-22T06:35:25.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:25 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:35:25.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.464+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7681a0b10 con 0x7fa768101fc0 2024-04-22T06:35:25.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.465+0000 7fa75effd640 1 -- 172.21.15.105:0/3849754986 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fa7640bf030 con 0x7fa768101fc0 2024-04-22T06:35:25.463 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:25.463 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-22T06:35:25.463 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 9 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 msgr2=0x7fa7540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 0x7fa7540a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa7580058c0 tx=0x7fa758005850 comp rx=0 tx=0).stop 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 msgr2=0x7fa768198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa764000c00 tx=0x7fa764004040 comp rx=0 tx=0).stop 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 shutdown_connections 2024-04-22T06:35:25.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7540a3bb0 0x7fa7540a6070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.466+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa768102980 0x7fa76819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.467+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa768101fc0 0x7fa768198710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.467+0000 7fa770104640 1 --2- 172.21.15.105:0/3849754986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa768100e70 0x7fa7681981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.467+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 >> 172.21.15.105:0/3849754986 conn(0x7fa7680fc3e0 msgr2=0x7fa7680690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.467+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 shutdown_connections 2024-04-22T06:35:25.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:25.467+0000 7fa770104640 1 -- 172.21.15.105:0/3849754986 wait complete. 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:25.610 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:25.611 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: Deploying daemon mds.i on smithi177 2024-04-22T06:35:25.611 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:25.611 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:boot 2024-04-22T06:35:25.611 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: fsmap 8 up:standby 2024-04-22T06:35:25.611 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:25 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:25.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: Deploying daemon mds.i on smithi177 2024-04-22T06:35:25.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:25.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:boot 2024-04-22T06:35:25.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: fsmap 8 up:standby 2024-04-22T06:35:25.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:25 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:35:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3849754986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:26.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 systemd[1]: Starting Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3849754986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:26.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3849754986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:26.858 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 podman[61894]: 2024-04-22 06:35:26.508450069 +0000 UTC m=+0.018178999 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:26.858 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 podman[61894]: 2024-04-22 06:35:26.615239785 +0000 UTC m=+0.124968730 container create 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:35:26.858 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 podman[61894]: 2024-04-22 06:35:26.8343073 +0000 UTC m=+0.344036202 container init 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T06:35:26.858 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 podman[61894]: 2024-04-22 06:35:26.836921905 +0000 UTC m=+0.346650808 container start 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-22T06:35:27.240 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 ceph-mds[61913]: -- 172.21.15.177:0/1033805420 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562af99b1a40 con 0x562af9985000 2024-04-22T06:35:27.240 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: starting mds.i at 2024-04-22T06:35:27.240 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 bash[61894]: 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 2024-04-22T06:35:27.240 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:35:26 smithi177 systemd[1]: Started Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:27.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:27.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- 172.21.15.105:0/2540907297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4101fc0 msgr2=0x7f4aa4102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- 172.21.15.105:0/2540907297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4102440 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009900 tx=0x7f4a9402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- 172.21.15.105:0/2540907297 shutdown_connections 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- 172.21.15.105:0/2540907297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4102980 0x7f4aa41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- 172.21.15.105:0/2540907297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4102440 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- 172.21.15.105:0/2540907297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- 172.21.15.105:0/2540907297 >> 172.21.15.105:0/2540907297 conn(0x7f4aa40fc3e0 msgr2=0x7f4aa40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- 172.21.15.105:0/2540907297 shutdown_connections 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- 172.21.15.105:0/2540907297 wait complete. 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 Processor -- start 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- start start 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410ca20 con 0x7f4aa4102980 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410c8a0 con 0x7f4aa4100e70 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.582+0000 7f4aabefb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410cba0 con 0x7f4aa4101fc0 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38988/0 (socket says 172.21.15.105:38988) 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 -- 172.21.15.105:0/561229322 learned_addr learned my addr 172.21.15.105:0/561229322 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aa9c97640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aa9496640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 -- 172.21.15.105:0/561229322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 msgr2=0x7f4aa4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 -- 172.21.15.105:0/561229322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 msgr2=0x7f4aa4198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 -- 172.21.15.105:0/561229322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa419fe70 con 0x7f4aa4102980 2024-04-22T06:35:27.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aa9c97640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aa9496640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aaa498640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000b970 tx=0x7f4aa000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0019070 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa0014440 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa41a0160 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00075d0 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.583+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41a06e0 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.584+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa41a09a0 con 0x7f4aa4102980 2024-04-22T06:35:27.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.585+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4aa001c070 con 0x7f4aa4102980 2024-04-22T06:35:27.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.586+0000 7f4a9affd640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 0x7f4a900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:27.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.586+0000 7f4aa9c97640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 0x7f4a900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:27.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.586+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f4aa00cfad0 con 0x7f4aa4102980 2024-04-22T06:35:27.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.586+0000 7f4aa9c97640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 0x7f4a900a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009c20 tx=0x7f4a9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:27.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.589+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4aa009c950 con 0x7f4aa4102980 2024-04-22T06:35:27.644 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:27.645 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:27 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:27.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:27 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:27.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:27 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:27.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.816+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4aa41a0b70 con 0x7f4aa4102980 2024-04-22T06:35:27.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.817+0000 7f4a9affd640 1 -- 172.21.15.105:0/561229322 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f4aa009c250 con 0x7f4aa4102980 2024-04-22T06:35:27.815 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:27.815 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:35:27.816 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 10 2024-04-22T06:35:27.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.818+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 msgr2=0x7f4a900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.818+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 0x7f4a900a6070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009c20 tx=0x7f4a9c005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.818+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 msgr2=0x7f4aa419f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.818+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000b970 tx=0x7f4aa000be30 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 shutdown_connections 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4a900a3bb0 0x7f4a900a6070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4aa4102980 0x7f4aa419f800 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4aa4101fc0 0x7f4aa4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 --2- 172.21.15.105:0/561229322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4aa4100e70 0x7f4aa4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 >> 172.21.15.105:0/561229322 conn(0x7f4aa40fc3e0 msgr2=0x7f4aa40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 shutdown_connections 2024-04-22T06:35:27.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:27.819+0000 7f4aabefb640 1 -- 172.21.15.105:0/561229322 wait complete. 2024-04-22T06:35:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:28 smithi177 ceph-mon[31346]: Deploying daemon mds.g on smithi105 2024-04-22T06:35:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:28 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:boot 2024-04-22T06:35:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:28 smithi177 ceph-mon[31346]: fsmap 9 up:standby 2024-04-22T06:35:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:28 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:35:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/561229322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:28 smithi105 ceph-mon[27916]: Deploying daemon mds.g on smithi105 2024-04-22T06:35:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:28 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:boot 2024-04-22T06:35:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:28 smithi105 ceph-mon[27916]: fsmap 9 up:standby 2024-04-22T06:35:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:28 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:35:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/561229322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:28.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:28 smithi112 ceph-mon[31236]: Deploying daemon mds.g on smithi105 2024-04-22T06:35:28.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:28 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:boot 2024-04-22T06:35:28.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:28 smithi112 ceph-mon[31236]: fsmap 9 up:standby 2024-04-22T06:35:28.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:28 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:35:28.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/561229322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:29.384 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 systemd[1]: Starting Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:29.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:29 smithi177 ceph-mon[31346]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:29.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:29 smithi105 ceph-mon[27916]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:29.668 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 podman[69822]: 2024-04-22 06:35:29.386064505 +0000 UTC m=+0.105972874 container create 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:35:29.668 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 podman[69822]: 2024-04-22 06:35:29.298072029 +0000 UTC m=+0.017980398 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:29.668 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 podman[69822]: 2024-04-22 06:35:29.605412215 +0000 UTC m=+0.325320581 container init 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:35:29.668 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 podman[69822]: 2024-04-22 06:35:29.608017416 +0000 UTC m=+0.327925792 container start 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:29.669 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 ceph-mds[69841]: -- 172.21.15.105:0/214277134 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56246671fa40 con 0x5624666f3000 2024-04-22T06:35:29.669 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: starting mds.g at 2024-04-22T06:35:29.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:29 smithi112 ceph-mon[31236]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:29.828 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:29.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- 172.21.15.105:0/4179074087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc101fc0 msgr2=0x7fb6fc102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 --2- 172.21.15.105:0/4179074087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc102440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009900 tx=0x7fb6f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- 172.21.15.105:0/4179074087 shutdown_connections 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 --2- 172.21.15.105:0/4179074087 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc102980 0x7fb6fc1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 --2- 172.21.15.105:0/4179074087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc102440 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 --2- 172.21.15.105:0/4179074087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc100e70 0x7fb6fc101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:29.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- 172.21.15.105:0/4179074087 >> 172.21.15.105:0/4179074087 conn(0x7fb6fc0fc3e0 msgr2=0x7fb6fc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- 172.21.15.105:0/4179074087 shutdown_connections 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- 172.21.15.105:0/4179074087 wait complete. 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 Processor -- start 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.936+0000 7fb700aba640 1 -- start start 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 0x7fb6fc197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:29.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10ca20 con 0x7fb6fc102980 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10c8a0 con 0x7fb6fc100e70 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc10cba0 con 0x7fb6fc101fc0 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39020/0 (socket says 172.21.15.105:39020) 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 -- 172.21.15.105:0/2018004871 learned_addr learned my addr 172.21.15.105:0/2018004871 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:29.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6f9d9b640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fa59c640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 0x7fb6fc197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 -- 172.21.15.105:0/2018004871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc101fc0 msgr2=0x7fb6fc197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 -- 172.21.15.105:0/2018004871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 msgr2=0x7fb6fc197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 0x7fb6fc197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:29.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 -- 172.21.15.105:0/2018004871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc19ed50 con 0x7fb6fc102980 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fa59c640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 0x7fb6fc197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6fad9d640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400cad0 tx=0x7fb6f400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400ae20 con 0x7fb6fc102980 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f4005020 con 0x7fb6fc102980 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc19efe0 con 0x7fb6fc102980 2024-04-22T06:35:29.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400f6b0 con 0x7fb6fc102980 2024-04-22T06:35:29.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.937+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc19f530 con 0x7fb6fc102980 2024-04-22T06:35:29.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.938+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc108270 con 0x7fb6fc102980 2024-04-22T06:35:29.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.940+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb6f40051d0 con 0x7fb6fc102980 2024-04-22T06:35:29.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.940+0000 7fb6eb7fe640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 0x7fb6e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:29.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.941+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fb6f40d03a0 con 0x7fb6fc102980 2024-04-22T06:35:29.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.941+0000 7fb6fa59c640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 0x7fb6e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:29.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.941+0000 7fb6fa59c640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 0x7fb6e40a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:29.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:29.944+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6f4014030 con 0x7fb6fc102980 2024-04-22T06:35:30.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.167+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6fc02d5c0 con 0x7fb6fc102980 2024-04-22T06:35:30.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.168+0000 7fb6eb7fe640 1 -- 172.21.15.105:0/2018004871 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7fb6f409d200 con 0x7fb6fc102980 2024-04-22T06:35:30.167 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:30.168 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:35:30.168 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 10 2024-04-22T06:35:30.169 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 bash[69822]: 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 2024-04-22T06:35:30.169 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:35:29 smithi105 systemd[1]: Started Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 msgr2=0x7fb6e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 0x7fb6e40a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec006fd0 tx=0x7fb6ec008040 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 msgr2=0x7fb6fc19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400cad0 tx=0x7fb6f400cf90 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 shutdown_connections 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb6e40a3bb0 0x7fb6e40a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6fc102980 0x7fb6fc19e6e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6fc101fc0 0x7fb6fc197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 --2- 172.21.15.105:0/2018004871 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6fc100e70 0x7fb6fc197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 >> 172.21.15.105:0/2018004871 conn(0x7fb6fc0fc3e0 msgr2=0x7fb6fc105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 shutdown_connections 2024-04-22T06:35:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:30.170+0000 7fb700aba640 1 -- 172.21.15.105:0/2018004871 wait complete. 2024-04-22T06:35:31.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: Deploying daemon mds.a on smithi105 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2018004871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:boot 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: fsmap 10 up:standby 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:35:31.164 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:30 smithi105 ceph-mon[27916]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: Deploying daemon mds.a on smithi105 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2018004871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:boot 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: fsmap 10 up:standby 2024-04-22T06:35:31.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:35:31.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:30 smithi112 ceph-mon[31236]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: Deploying daemon mds.a on smithi105 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2018004871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:boot 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: fsmap 10 up:standby 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:35:31.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:30 smithi177 ceph-mon[31346]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:31.979 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:31 smithi105 systemd[1]: Starting Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:32.181 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:32.255 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 podman[70170]: 2024-04-22 06:35:32.143571933 +0000 UTC m=+0.109436430 container create 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:35:32.255 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 podman[70170]: 2024-04-22 06:35:32.052240298 +0000 UTC m=+0.018104808 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/384180383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4102a50 msgr2=0x7fd8e4104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/384180383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4102a50 0x7fd8e4104e40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009900 tx=0x7fd8e002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/384180383 shutdown_connections 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/384180383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4107cb0 0x7fd8e410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/384180383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 0x7fd8e4107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/384180383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4102a50 0x7fd8e4104e40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.287+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/384180383 >> 172.21.15.105:0/384180383 conn(0x7fd8e40fc3e0 msgr2=0x7fd8e40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:32.290 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/384180383 shutdown_connections 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/384180383 wait complete. 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 Processor -- start 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- start start 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4102a50 0x7fd8e41023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 0x7fd8e4100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e410dab0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e410d930 con 0x7fd8e4102a50 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ebdd5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e410dc30 con 0x7fd8e4105380 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ea372640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:32.291 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ea372640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39038/0 (socket says 172.21.15.105:39038) 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.288+0000 7fd8ea372640 1 -- 172.21.15.105:0/1216583038 learned_addr learned my addr 172.21.15.105:0/1216583038 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8e9b71640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4102a50 0x7fd8e41023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8e9370640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 0x7fd8e4100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 -- 172.21.15.105:0/1216583038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 msgr2=0x7fd8e4100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 0x7fd8e4100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 -- 172.21.15.105:0/1216583038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4102a50 msgr2=0x7fd8e41023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4102a50 0x7fd8e41023c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 -- 172.21.15.105:0/1216583038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e41017d0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ea372640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00cce0 tx=0x7fd8dc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc00f450 con 0x7fd8e4107cb0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e41a22b0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8dc00faa0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc018b70 con 0x7fd8e4107cb0 2024-04-22T06:35:32.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.289+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e41a2850 con 0x7fd8e4107cb0 2024-04-22T06:35:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.290+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e40fded0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.292+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fd8dc00f5f0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.292+0000 7fd8daffd640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 0x7fd8d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:32.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.293+0000 7fd8e9b71640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 0x7fd8d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.293+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fd8dc0d04b0 con 0x7fd8e4107cb0 2024-04-22T06:35:32.295 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.293+0000 7fd8e9b71640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 0x7fd8d00a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009230 tx=0x7fd8e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:32.298 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.300+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8dc014030 con 0x7fd8e4107cb0 2024-04-22T06:35:32.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.530+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8e4062b50 con 0x7fd8e4107cb0 2024-04-22T06:35:32.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.530+0000 7fd8daffd640 1 -- 172.21.15.105:0/1216583038 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7fd8dc09d330 con 0x7fd8e4107cb0 2024-04-22T06:35:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:32.529 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:35:32.530 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 12 2024-04-22T06:35:32.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 msgr2=0x7fd8d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:32.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 0x7fd8d00a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009230 tx=0x7fd8e0005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 msgr2=0x7fd8e4100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:32.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00cce0 tx=0x7fd8dc00a620 comp rx=0 tx=0).stop 2024-04-22T06:35:32.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 shutdown_connections 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd8d00a3bb0 0x7fd8d00a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8e4107cb0 0x7fd8e4100f40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd8e4105380 0x7fd8e4100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 --2- 172.21.15.105:0/1216583038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8e4102a50 0x7fd8e41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.532+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 >> 172.21.15.105:0/1216583038 conn(0x7fd8e40fc3e0 msgr2=0x7fd8e4103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.533+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 shutdown_connections 2024-04-22T06:35:32.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:32.533+0000 7fd8ebdd5640 1 -- 172.21.15.105:0/1216583038 wait complete. 2024-04-22T06:35:32.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 podman[70170]: 2024-04-22 06:35:32.379585293 +0000 UTC m=+0.345449793 container init 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:32.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 podman[70170]: 2024-04-22 06:35:32.382840378 +0000 UTC m=+0.348704882 container start 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:35:32.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 ceph-mds[70236]: -- 172.21.15.105:0/1768603786 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589f8ab3a40 con 0x5589f8a87000 2024-04-22T06:35:32.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: starting mds.a at 2024-04-22T06:35:32.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 bash[70170]: 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec 2024-04-22T06:35:32.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:35:32 smithi105 systemd[1]: Started Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:boot 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: fsmap 11 up:standby 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1216583038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:33.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:33.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:33.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:33.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: Deploying daemon mds.b on smithi112 2024-04-22T06:35:33.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:33 smithi177 ceph-mon[31346]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:boot 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: fsmap 11 up:standby 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1216583038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:33.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: Deploying daemon mds.b on smithi112 2024-04-22T06:35:33.853 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:33 smithi112 ceph-mon[31236]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:boot 2024-04-22T06:35:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: fsmap 11 up:standby 2024-04-22T06:35:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:35:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1216583038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: Deploying daemon mds.b on smithi112 2024-04-22T06:35:33.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:33 smithi105 ceph-mon[27916]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:34.543 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:34.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 -- 172.21.15.105:0/1942634810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101fc0 msgr2=0x7f2d74102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/1942634810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101fc0 0x7f2d74102440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009900 tx=0x7f2d6802bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:34.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 -- 172.21.15.105:0/1942634810 shutdown_connections 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/1942634810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74102980 0x7f2d741092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/1942634810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74101fc0 0x7f2d74102440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/1942634810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74100e70 0x7f2d74101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 -- 172.21.15.105:0/1942634810 >> 172.21.15.105:0/1942634810 conn(0x7f2d740fc3e0 msgr2=0x7f2d740fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 -- 172.21.15.105:0/1942634810 shutdown_connections 2024-04-22T06:35:34.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.657+0000 7f2d7a37e640 1 -- 172.21.15.105:0/1942634810 wait complete. 2024-04-22T06:35:34.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.658+0000 7f2d7a37e640 1 Processor -- start 2024-04-22T06:35:34.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.658+0000 7f2d7a37e640 1 -- start start 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 0x7f2d7419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74102980 0x7f2d7419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410c9d0 con 0x7f2d74100e70 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410c850 con 0x7f2d74102980 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7410cb50 con 0x7f2d74101fc0 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48488/0 (socket says 172.21.15.105:48488) 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 -- 172.21.15.105:0/4166584986 learned_addr learned my addr 172.21.15.105:0/4166584986 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7891b640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74102980 0x7f2d7419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d737fe640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 0x7f2d7419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 -- 172.21.15.105:0/4166584986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 msgr2=0x7f2d7419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 0x7f2d7419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 -- 172.21.15.105:0/4166584986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74102980 msgr2=0x7f2d7419f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74102980 0x7f2d7419f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 -- 172.21.15.105:0/4166584986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7419fe70 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d737fe640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 0x7f2d7419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d73fff640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400d950 tx=0x7f2d6400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64017070 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d64009c30 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d741a0100 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400f620 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.659+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d741a0620 con 0x7f2d74100e70 2024-04-22T06:35:34.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.661+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d741082a0 con 0x7f2d74100e70 2024-04-22T06:35:34.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.662+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2d64010460 con 0x7f2d74100e70 2024-04-22T06:35:34.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.662+0000 7f2d717fa640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 0x7f2d5c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:34.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.663+0000 7f2d737fe640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 0x7f2d5c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:34.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.663+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f2d640cfb90 con 0x7f2d74100e70 2024-04-22T06:35:34.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.663+0000 7f2d737fe640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 0x7f2d5c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006010 tx=0x7f2d6802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:34.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.667+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d6409ca10 con 0x7f2d74100e70 2024-04-22T06:35:34.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.883+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d74062b50 con 0x7f2d74100e70 2024-04-22T06:35:34.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.883+0000 7f2d717fa640 1 -- 172.21.15.105:0/4166584986 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7f2d64009de0 con 0x7f2d74100e70 2024-04-22T06:35:34.882 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:34.882 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-22T06:35:34.882 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 12 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 msgr2=0x7f2d5c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 0x7f2d5c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006010 tx=0x7f2d6802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 msgr2=0x7f2d74069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400d950 tx=0x7f2d6400de10 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 shutdown_connections 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2d5c0a3bb0 0x7f2d5c0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d74102980 0x7f2d7419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2d74101fc0 0x7f2d7419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 --2- 172.21.15.105:0/4166584986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d74100e70 0x7f2d74069c50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 >> 172.21.15.105:0/4166584986 conn(0x7f2d740fc3e0 msgr2=0x7f2d74105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:34.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 shutdown_connections 2024-04-22T06:35:34.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:34.885+0000 7f2d7a37e640 1 -- 172.21.15.105:0/4166584986 wait complete. 2024-04-22T06:35:34.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:34 smithi112 systemd[1]: Starting Ceph mds.b for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:35:35.254 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:34 smithi112 podman[61038]: 2024-04-22 06:35:34.888147241 +0000 UTC m=+0.017889196 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:35:35.254 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 podman[61038]: 2024-04-22 06:35:35.054577897 +0000 UTC m=+0.184319838 container create 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True) 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 podman[61038]: 2024-04-22 06:35:35.256881766 +0000 UTC m=+0.386623707 container init 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 podman[61038]: 2024-04-22 06:35:35.259327568 +0000 UTC m=+0.389069512 container start 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mds[61057]: -- 172.21.15.112:0/2744715545 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5597406a3a40 con 0x559740676800 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: starting mds.b at 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 bash[61038]: 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 2024-04-22T06:35:35.674 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:35 smithi112 systemd[1]: Started Ceph mds.b for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4166584986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.675 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:35 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4166584986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:35.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:35 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4166584986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:36.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:36.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:36.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:35 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:36.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:36 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:boot 2024-04-22T06:35:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:36 smithi112 ceph-mon[31236]: fsmap 12 up:standby 2024-04-22T06:35:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:36 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:35:36.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:36 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:boot 2024-04-22T06:35:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:36 smithi177 ceph-mon[31346]: fsmap 12 up:standby 2024-04-22T06:35:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:36 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 -- 172.21.15.105:0/3163404923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 msgr2=0x7f613810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 --2- 172.21.15.105:0/3163404923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613810a0a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6130009ca0 tx=0x7f613002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 -- 172.21.15.105:0/3163404923 shutdown_connections 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 --2- 172.21.15.105:0/3163404923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613810a0a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 --2- 172.21.15.105:0/3163404923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138105380 0x7f6138107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 --2- 172.21.15.105:0/3163404923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138102a50 0x7f6138104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 -- 172.21.15.105:0/3163404923 >> 172.21.15.105:0/3163404923 conn(0x7f61380fc3e0 msgr2=0x7f61380fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 -- 172.21.15.105:0/3163404923 shutdown_connections 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.007+0000 7f613e8b3640 1 -- 172.21.15.105:0/3163404923 wait complete. 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 Processor -- start 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 -- start start 2024-04-22T06:35:37.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 0x7f61381981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138105380 0x7f6138198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f613810dab0 con 0x7f6138107cb0 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f613810d930 con 0x7f6138105380 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613e8b3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f613810dc30 con 0x7f6138102a50 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48500/0 (socket says 172.21.15.105:48500) 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f6137fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 0x7f61381981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 -- 172.21.15.105:0/2964876044 learned_addr learned my addr 172.21.15.105:0/2964876044 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f61377fe640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138105380 0x7f6138198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 -- 172.21.15.105:0/2964876044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 msgr2=0x7f61381981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 0x7f61381981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 -- 172.21.15.105:0/2964876044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138105380 msgr2=0x7f6138198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138105380 0x7f6138198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 -- 172.21.15.105:0/2964876044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613819fd90 con 0x7f6138107cb0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f6137fff640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 0x7f61381981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.008+0000 7f613ce50640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f613002c700 tx=0x7f61300346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.009+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130041020 con 0x7f6138107cb0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.009+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6130034ca0 con 0x7f6138107cb0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.009+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381a0020 con 0x7f6138107cb0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.009+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6130039040 con 0x7f6138107cb0 2024-04-22T06:35:37.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.009+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61381a0510 con 0x7f6138107cb0 2024-04-22T06:35:37.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.010+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61380fded0 con 0x7f6138107cb0 2024-04-22T06:35:37.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.011+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6130044020 con 0x7f6138107cb0 2024-04-22T06:35:37.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.011+0000 7f61357fa640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 0x7f61200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.012+0000 7f6137fff640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 0x7f61200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.012+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f61300f2180 con 0x7f6138107cb0 2024-04-22T06:35:37.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.012+0000 7f6137fff640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 0x7f61200a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6128009fc0 tx=0x7f6128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.015+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f613004b020 con 0x7f6138107cb0 2024-04-22T06:35:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:36 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:boot 2024-04-22T06:35:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:36 smithi105 ceph-mon[27916]: fsmap 12 up:standby 2024-04-22T06:35:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:36 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:35:37.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.284+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f613802d5c0 con 0x7f6138107cb0 2024-04-22T06:35:37.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.284+0000 7f61357fa640 1 -- 172.21.15.105:0/2964876044 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f61300bf080 con 0x7f6138107cb0 2024-04-22T06:35:37.283 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:37.283 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24461,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-22T06:35:37.283 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 13 2024-04-22T06:35:37.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 msgr2=0x7f61200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 0x7f61200a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6128009fc0 tx=0x7f6128005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:37.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 msgr2=0x7f613819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f613002c700 tx=0x7f61300346d0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 shutdown_connections 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f61200a3bb0 0x7f61200a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6138107cb0 0x7f613819f720 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138105380 0x7f6138198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 --2- 172.21.15.105:0/2964876044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6138102a50 0x7f61381981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.286+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 >> 172.21.15.105:0/2964876044 conn(0x7f61380fc3e0 msgr2=0x7f6138103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.287+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 shutdown_connections 2024-04-22T06:35:37.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.287+0000 7f613e8b3640 1 -- 172.21.15.105:0/2964876044 wait complete. 2024-04-22T06:35:37.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:35:37.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 -- 172.21.15.105:0/1340492840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec101fd0 msgr2=0x7f41ec102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 --2- 172.21.15.105:0/1340492840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec101fd0 0x7f41ec102450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009900 tx=0x7f41e002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:37.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 -- 172.21.15.105:0/1340492840 shutdown_connections 2024-04-22T06:35:37.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 --2- 172.21.15.105:0/1340492840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec102990 0x7f41ec1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 --2- 172.21.15.105:0/1340492840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec101fd0 0x7f41ec102450 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 --2- 172.21.15.105:0/1340492840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec100e80 0x7f41ec1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 -- 172.21.15.105:0/1340492840 >> 172.21.15.105:0/1340492840 conn(0x7f41ec0fc410 msgr2=0x7f41ec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 -- 172.21.15.105:0/1340492840 shutdown_connections 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.397+0000 7f41f404e640 1 -- 172.21.15.105:0/1340492840 wait complete. 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 Processor -- start 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- start start 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec101fd0 0x7f41ec198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 0x7f41ec19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec10ca20 con 0x7f41ec100e80 2024-04-22T06:35:37.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec10c8a0 con 0x7f41ec102990 2024-04-22T06:35:37.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec10cba0 con 0x7f41ec101fd0 2024-04-22T06:35:37.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48514/0 (socket says 172.21.15.105:48514) 2024-04-22T06:35:37.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 -- 172.21.15.105:0/3725246747 learned_addr learned my addr 172.21.15.105:0/3725246747 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:37.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f15e9640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec101fd0 0x7f41ec198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 -- 172.21.15.105:0/3725246747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec101fd0 msgr2=0x7f41ec198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec101fd0 0x7f41ec198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f25eb640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 0x7f41ec19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 -- 172.21.15.105:0/3725246747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 msgr2=0x7f41ec19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 0x7f41ec19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 -- 172.21.15.105:0/3725246747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec19fe00 con 0x7f41ec100e80 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f25eb640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 0x7f41ec19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f1dea640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41e400db80 tx=0x7f41e400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4017070 con 0x7f41ec100e80 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1a00f0 con 0x7f41ec100e80 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41e4003b70 con 0x7f41ec100e80 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e40106c0 con 0x7f41ec100e80 2024-04-22T06:35:37.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.398+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec1a0610 con 0x7f41ec100e80 2024-04-22T06:35:37.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.400+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec1a0870 con 0x7f41ec100e80 2024-04-22T06:35:37.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.400+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f41e4004020 con 0x7f41ec100e80 2024-04-22T06:35:37.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.401+0000 7f41eaffd640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 0x7f41d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.401+0000 7f41f15e9640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 0x7f41d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.401+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f41e40d0000 con 0x7f41ec100e80 2024-04-22T06:35:37.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.402+0000 7f41f15e9640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 0x7f41d40a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41e002c240 tx=0x7f41e002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.406+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41e409ce80 con 0x7f41ec100e80 2024-04-22T06:35:37.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.585+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41ec1a0a40 con 0x7f41ec100e80 2024-04-22T06:35:37.585 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.586+0000 7f41eaffd640 1 -- 172.21.15.105:0/3725246747 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+18502 (secure 0 0 0) 0x7f41e409c780 con 0x7f41ec100e80 2024-04-22T06:35:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:37.585 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":85,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:34:51.679181+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:37.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.588+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 msgr2=0x7f41d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.588+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 0x7f41d40a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41e002c240 tx=0x7f41e002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 msgr2=0x7f41ec1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41e400db80 tx=0x7f41e400b630 comp rx=0 tx=0).stop 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 shutdown_connections 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41d40a3bb0 0x7f41d40a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41ec102990 0x7f41ec19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f41ec101fd0 0x7f41ec198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 --2- 172.21.15.105:0/3725246747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41ec100e80 0x7f41ec1981e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 >> 172.21.15.105:0/3725246747 conn(0x7f41ec0fc410 msgr2=0x7f41ec0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 shutdown_connections 2024-04-22T06:35:37.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.589+0000 7f41f404e640 1 -- 172.21.15.105:0/3725246747 wait complete. 2024-04-22T06:35:37.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:35:37.599 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:35:37.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 -- 172.21.15.105:0/3370571951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728101fa0 msgr2=0x7f07281023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 --2- 172.21.15.105:0/3370571951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728101fa0 0x7f07281023c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0718009900 tx=0x7f071802bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:37.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 -- 172.21.15.105:0/3370571951 shutdown_connections 2024-04-22T06:35:37.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 --2- 172.21.15.105:0/3370571951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728103b50 0x7f072810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 --2- 172.21.15.105:0/3370571951 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103190 0x7f0728103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 --2- 172.21.15.105:0/3370571951 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728101fa0 0x7f07281023c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 -- 172.21.15.105:0/3370571951 >> 172.21.15.105:0/3370571951 conn(0x7f07280fd530 msgr2=0x7f07280ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 -- 172.21.15.105:0/3370571951 shutdown_connections 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 -- 172.21.15.105:0/3370571951 wait complete. 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.702+0000 7f072d77e640 1 Processor -- start 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- start start 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728101fa0 0x7f0728198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103b50 0x7f072819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f072810dae0 con 0x7f0728103190 2024-04-22T06:35:37.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f072810d960 con 0x7f0728101fa0 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f072810dc60 con 0x7f0728103b50 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48532/0 (socket says 172.21.15.105:48532) 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 -- 172.21.15.105:0/4138655386 learned_addr learned my addr 172.21.15.105:0/4138655386 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07277fe640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103b50 0x7f072819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f0726ffd640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728101fa0 0x7f0728198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 -- 172.21.15.105:0/4138655386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103b50 msgr2=0x7f072819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103b50 0x7f072819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 -- 172.21.15.105:0/4138655386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728101fa0 msgr2=0x7f0728198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728101fa0 0x7f0728198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 -- 172.21.15.105:0/4138655386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072819fe50 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f07267fc640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e9f0 tx=0x7f071c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c00cd80 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07281a0140 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f071c0054d0 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c010690 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.703+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07281a0640 con 0x7f0728103190 2024-04-22T06:35:37.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.704+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0728109480 con 0x7f0728103190 2024-04-22T06:35:37.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.706+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f071c004e50 con 0x7f0728103190 2024-04-22T06:35:37.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.706+0000 7f070ffff640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 0x7f07100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:37.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.706+0000 7f0726ffd640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 0x7f07100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:37.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.706+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f071c0d0630 con 0x7f0728103190 2024-04-22T06:35:37.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.707+0000 7f0726ffd640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 0x7f07100a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0718006010 tx=0x7f0718005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:37.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.710+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f071c053ff0 con 0x7f0728103190 2024-04-22T06:35:37.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.892+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f072802d810 con 0x7f0728103190 2024-04-22T06:35:37.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.892+0000 7f070ffff640 1 -- 172.21.15.105:0/4138655386 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f071c09d4b0 con 0x7f0728103190 2024-04-22T06:35:37.891 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 msgr2=0x7f07100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 0x7f07100a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0718006010 tx=0x7f0718005be0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 msgr2=0x7f0728198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f071c00e9f0 tx=0x7f071c00edf0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 shutdown_connections 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f07100a3bb0 0x7f07100a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0728103b50 0x7f072819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0728103190 0x7f0728198770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 --2- 172.21.15.105:0/4138655386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0728101fa0 0x7f0728198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 >> 172.21.15.105:0/4138655386 conn(0x7f07280fd530 msgr2=0x7f0728106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 shutdown_connections 2024-04-22T06:35:37.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:37.895+0000 7f072d77e640 1 -- 172.21.15.105:0/4138655386 wait complete. 2024-04-22T06:35:37.903 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:37.903 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.903 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.903 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.904 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.904 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2964876044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:37.904 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.904 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.904 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:37 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3725246747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:37.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-22T06:35:37.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2964876044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:37 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3725246747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2964876044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:37.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:37 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3725246747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 -- 172.21.15.105:0/377517803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 msgr2=0x7f4188101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 --2- 172.21.15.105:0/377517803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f4188101280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4180009ca0 tx=0x7f418002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 -- 172.21.15.105:0/377517803 shutdown_connections 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 --2- 172.21.15.105:0/377517803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102a00 0x7f4188109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 --2- 172.21.15.105:0/377517803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102040 0x7f41881024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 --2- 172.21.15.105:0/377517803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f4188101280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 -- 172.21.15.105:0/377517803 >> 172.21.15.105:0/377517803 conn(0x7f41880fc440 msgr2=0x7f41880fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 -- 172.21.15.105:0/377517803 shutdown_connections 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.007+0000 7f418d784640 1 -- 172.21.15.105:0/377517803 wait complete. 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 Processor -- start 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 -- start start 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 0x7f418819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102a00 0x7f418819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f418810c9f0 con 0x7f4188100e60 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f418810c870 con 0x7f4188102a00 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f418d784640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f418810cb70 con 0x7f4188102040 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f4186ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f4186ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48548/0 (socket says 172.21.15.105:48548) 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f4186ffd640 1 -- 172.21.15.105:0/3418153096 learned_addr learned my addr 172.21.15.105:0/3418153096 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:38.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f41877fe640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102a00 0x7f418819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.008+0000 7f41867fc640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 0x7f418819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 -- 172.21.15.105:0/3418153096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 msgr2=0x7f418819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 0x7f418819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 -- 172.21.15.105:0/3418153096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102a00 msgr2=0x7f418819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102a00 0x7f418819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 -- 172.21.15.105:0/3418153096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418819fe10 con 0x7f4188100e60 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f41867fc640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 0x7f418819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f4186ffd640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4180005e30 tx=0x7f41800346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180041020 con 0x7f4188100e60 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41881a00a0 con 0x7f4188100e60 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4180034ca0 con 0x7f4188100e60 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180039040 con 0x7f4188100e60 2024-04-22T06:35:38.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.009+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41881a05e0 con 0x7f4188100e60 2024-04-22T06:35:38.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.010+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4188108320 con 0x7f4188100e60 2024-04-22T06:35:38.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.011+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4180044020 con 0x7f4188100e60 2024-04-22T06:35:38.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.012+0000 7f416ffff640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 0x7f41700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.012+0000 7f41867fc640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 0x7f41700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.012+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f41800f2120 con 0x7f4188100e60 2024-04-22T06:35:38.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.013+0000 7f41867fc640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 0x7f41700a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f417c005d00 tx=0x7f417c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.016+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f418004b020 con 0x7f4188100e60 2024-04-22T06:35:38.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.203+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f418802d740 con 0x7f4188100e60 2024-04-22T06:35:38.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.203+0000 7f416ffff640 1 -- 172.21.15.105:0/3418153096 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) ==== 84+0+65 (secure 0 0 0) 0x7f41800befa0 con 0x7f4188100e60 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout:[ 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "poolnum": 1, 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "poolname": ".mgr" 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:35:38.202 INFO:teuthology.orchestra.run.smithi105.stdout:] 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 msgr2=0x7f41700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 0x7f41700a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f417c005d00 tx=0x7f417c005c90 comp rx=0 tx=0).stop 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 msgr2=0x7f418806ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4180005e30 tx=0x7f41800346e0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 shutdown_connections 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f41700a3bb0 0x7f41700a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4188102a00 0x7f418819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4188102040 0x7f418819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 --2- 172.21.15.105:0/3418153096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4188100e60 0x7f418806ae80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 >> 172.21.15.105:0/3418153096 conn(0x7f41880fc440 msgr2=0x7f41881054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 shutdown_connections 2024-04-22T06:35:38.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.205+0000 7f418d784640 1 -- 172.21.15.105:0/3418153096 wait complete. 2024-04-22T06:35:38.215 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-22T06:35:38.215 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 -- 172.21.15.105:0/2363013988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 msgr2=0x7f0d74102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/2363013988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74102460 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d70009900 tx=0x7f0d7002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 -- 172.21.15.105:0/2363013988 shutdown_connections 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/2363013988 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d741029a0 0x7f0d741092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/2363013988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74102460 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/2363013988 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d74100e90 0x7f0d741012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 -- 172.21.15.105:0/2363013988 >> 172.21.15.105:0/2363013988 conn(0x7f0d740fc400 msgr2=0x7f0d740fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 -- 172.21.15.105:0/2363013988 shutdown_connections 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.321+0000 7f0d7b55f640 1 -- 172.21.15.105:0/2363013988 wait complete. 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 Processor -- start 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 -- start start 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d74100e90 0x7f0d74197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 0x7f0d7419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410c9e0 con 0x7f0d74101fe0 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410c860 con 0x7f0d741029a0 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d7b55f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410cb60 con 0x7f0d74100e90 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48560/0 (socket says 172.21.15.105:48560) 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 -- 172.21.15.105:0/3054169424 learned_addr learned my addr 172.21.15.105:0/3054169424 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:38.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d79afc640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 0x7f0d7419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d792fb640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d74100e90 0x7f0d74197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 -- 172.21.15.105:0/3054169424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d74100e90 msgr2=0x7f0d74197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d74100e90 0x7f0d74197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 -- 172.21.15.105:0/3054169424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 msgr2=0x7f0d7419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 0x7f0d7419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d78afa640 1 -- 172.21.15.105:0/3054169424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7419ed60 con 0x7f0d74101fe0 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.322+0000 7f0d79afc640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 0x7f0d7419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d78afa640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000c00 tx=0x7f0d70004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70043020 con 0x7f0d74101fe0 2024-04-22T06:35:38.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d70004440 con 0x7f0d74101fe0 2024-04-22T06:35:38.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70003c90 con 0x7f0d74101fe0 2024-04-22T06:35:38.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7419eff0 con 0x7f0d74101fe0 2024-04-22T06:35:38.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.323+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d7419f450 con 0x7f0d74101fe0 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.324+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d74108290 con 0x7f0d74101fe0 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.325+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0d70046020 con 0x7f0d74101fe0 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.326+0000 7f0d6a7fc640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 0x7f0d600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.326+0000 7f0d792fb640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 0x7f0d600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.326+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f0d700f2500 con 0x7f0d74101fe0 2024-04-22T06:35:38.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.326+0000 7f0d792fb640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 0x7f0d600a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009bd0 tx=0x7f0d64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.329+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d7004d020 con 0x7f0d74101fe0 2024-04-22T06:35:38.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.530+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0d7402d5c0 con 0x7f0d74101fe0 2024-04-22T06:35:38.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.786+0000 7f0d6a7fc640 1 -- 172.21.15.105:0/3054169424 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) ==== 138+0+0 (secure 0 0 0) 0x7f0d700bf380 con 0x7f0d74101fe0 2024-04-22T06:35:38.785 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'cephfs_metadata' created 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 msgr2=0x7f0d600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 0x7f0d600a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009bd0 tx=0x7f0d64005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 msgr2=0x7f0d74197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000c00 tx=0x7f0d70004040 comp rx=0 tx=0).stop 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 shutdown_connections 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0d600a3bb0 0x7f0d600a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d741029a0 0x7f0d7419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d74101fe0 0x7f0d74197680 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 --2- 172.21.15.105:0/3054169424 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0d74100e90 0x7f0d74197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 >> 172.21.15.105:0/3054169424 conn(0x7f0d740fc400 msgr2=0x7f0d74105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.789+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 shutdown_connections 2024-04-22T06:35:38.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.790+0000 7f0d7b55f640 1 -- 172.21.15.105:0/3054169424 wait complete. 2024-04-22T06:35:38.802 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.911+0000 7fdf02d10640 1 -- 172.21.15.105:0/963385267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 msgr2=0x7fdefc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.911+0000 7fdf02d10640 1 --2- 172.21.15.105:0/963385267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc1092c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009ca0 tx=0x7fdef402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- 172.21.15.105:0/963385267 shutdown_connections 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- 172.21.15.105:0/963385267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc1092c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- 172.21.15.105:0/963385267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- 172.21.15.105:0/963385267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdefc100e90 0x7fdefc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- 172.21.15.105:0/963385267 >> 172.21.15.105:0/963385267 conn(0x7fdefc0fc400 msgr2=0x7fdefc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- 172.21.15.105:0/963385267 shutdown_connections 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- 172.21.15.105:0/963385267 wait complete. 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 Processor -- start 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- start start 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdefc100e90 0x7fdefc1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10c9d0 con 0x7fdefc1029a0 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10c850 con 0x7fdefc101fe0 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf02d10640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc10cb50 con 0x7fdefc100e90 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf012ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf012ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48572/0 (socket says 172.21.15.105:48572) 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdf012ad640 1 -- 172.21.15.105:0/2821531077 learned_addr learned my addr 172.21.15.105:0/2821531077 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.912+0000 7fdefbfff640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 -- 172.21.15.105:0/2821531077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdefc100e90 msgr2=0x7fdefc1970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdefc100e90 0x7fdefc1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 -- 172.21.15.105:0/2821531077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 msgr2=0x7fdefc197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 -- 172.21.15.105:0/2821531077 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc19ece0 con 0x7fdefc1029a0 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdefbfff640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:38.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf012ad640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdef402c700 tx=0x7fdef40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4041020 con 0x7fdefc1029a0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc19ef70 con 0x7fdefc1029a0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdef4034d60 con 0x7fdefc1029a0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4039040 con 0x7fdefc1029a0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.913+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc19f3e0 con 0x7fdefc1029a0 2024-04-22T06:35:38.916 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.914+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdefc108290 con 0x7fdefc1029a0 2024-04-22T06:35:38.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.915+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fdef4044020 con 0x7fdefc1029a0 2024-04-22T06:35:38.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.916+0000 7fdef9ffb640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 0x7fdeec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:38.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.916+0000 7fdf00aac640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 0x7fdeec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:38.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.916+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7497+0+0 (secure 0 0 0) 0x7fdef40f2510 con 0x7fdefc1029a0 2024-04-22T06:35:38.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.916+0000 7fdf00aac640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 0x7fdeec0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdef000ba80 tx=0x7fdef0005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:38.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:38.920+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdef404b020 con 0x7fdefc1029a0 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4138655386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:38.946 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3418153096' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:35:38.947 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:38 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:35:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:38.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:38.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:38.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4138655386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:38.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:38.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3418153096' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:35:38.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:38 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:35:38.967 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:38.967 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:38.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:38.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4138655386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:35:38.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:35:38.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3418153096' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-22T06:35:38.968 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-22T06:35:39.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.103+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdefc02d5c0 con 0x7fdefc1029a0 2024-04-22T06:35:39.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.788+0000 7fdef9ffb640 1 -- 172.21.15.105:0/2821531077 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) ==== 170+0+0 (secure 0 0 0) 0x7fdef40bf220 con 0x7fdefc1029a0 2024-04-22T06:35:39.786 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'cephfs_data' created 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 msgr2=0x7fdeec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 0x7fdeec0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdef000ba80 tx=0x7fdef0005ed0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 msgr2=0x7fdefc19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdef402c700 tx=0x7fdef40093f0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 shutdown_connections 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fdeec0a3bb0 0x7fdeec0a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdefc1029a0 0x7fdefc19e670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdefc101fe0 0x7fdefc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 --2- 172.21.15.105:0/2821531077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdefc100e90 0x7fdefc1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 >> 172.21.15.105:0/2821531077 conn(0x7fdefc0fc400 msgr2=0x7fdefc105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:39.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 shutdown_connections 2024-04-22T06:35:39.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.790+0000 7fdf02d10640 1 -- 172.21.15.105:0/2821531077 wait complete. 2024-04-22T06:35:39.808 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-22T06:35:39.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 -- 172.21.15.105:0/773379242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54301029a0 msgr2=0x7f54301092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 --2- 172.21.15.105:0/773379242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54301029a0 0x7f54301092c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5428009ca0 tx=0x7f542802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 -- 172.21.15.105:0/773379242 shutdown_connections 2024-04-22T06:35:39.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 --2- 172.21.15.105:0/773379242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54301029a0 0x7f54301092c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 --2- 172.21.15.105:0/773379242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 --2- 172.21.15.105:0/773379242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5430100e90 0x7f54301012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 -- 172.21.15.105:0/773379242 >> 172.21.15.105:0/773379242 conn(0x7f54300fc400 msgr2=0x7f54300fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 -- 172.21.15.105:0/773379242 shutdown_connections 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.909+0000 7f543666d640 1 -- 172.21.15.105:0/773379242 wait complete. 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 Processor -- start 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 -- start start 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54301029a0 0x7f543019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f543010db50 con 0x7f5430100e90 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f543010d9d0 con 0x7f5430101fe0 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f543666d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f543010dcd0 con 0x7f54301029a0 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f542ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f542ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48600/0 (socket says 172.21.15.105:48600) 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.910+0000 7f542ffff640 1 -- 172.21.15.105:0/3808596932 learned_addr learned my addr 172.21.15.105:0/3808596932 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f5434c0a640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54301029a0 0x7f543019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542f7fe640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 -- 172.21.15.105:0/3808596932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54301029a0 msgr2=0x7f543019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54301029a0 0x7f543019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 -- 172.21.15.105:0/3808596932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 msgr2=0x7f5430198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 -- 172.21.15.105:0/3808596932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543019fe70 con 0x7f5430100e90 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542f7fe640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542ffff640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f542000b4a0 tx=0x7f542000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420014070 con 0x7f5430100e90 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5420010ac0 con 0x7f5430100e90 2024-04-22T06:35:39.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54301a0160 con 0x7f5430100e90 2024-04-22T06:35:39.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420017c30 con 0x7f5430100e90 2024-04-22T06:35:39.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.911+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54301a0660 con 0x7f5430100e90 2024-04-22T06:35:39.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.912+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5430108210 con 0x7f5430100e90 2024-04-22T06:35:39.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.913+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f5420017070 con 0x7f5430100e90 2024-04-22T06:35:39.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.914+0000 7f542d7fa640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 0x7f54180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:39.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.914+0000 7f542f7fe640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 0x7f54180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:39.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.914+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7879+0+0 (secure 0 0 0) 0x7f54200cf690 con 0x7f5430100e90 2024-04-22T06:35:39.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.914+0000 7f542f7fe640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 0x7f54180a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5430199760 tx=0x7f542400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:39.917 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:39.918+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f542009c180 con 0x7f5430100e90 2024-04-22T06:35:39.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:39.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:39 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:39.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:39.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:39.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:39 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.133+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f543002d5c0 con 0x7f5430100e90 2024-04-22T06:35:40.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:40.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3054169424' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-22T06:35:40.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: osdmap e86: 12 total, 12 up, 12 in 2024-04-22T06:35:40.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-22T06:35:40.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:35:40.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:39 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:35:40.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.801+0000 7f542d7fa640 1 -- 172.21.15.105:0/3808596932 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f542009ba80 con 0x7f5430100e90 2024-04-22T06:35:40.800 INFO:teuthology.orchestra.run.smithi105.stderr:new fs with metadata pool 2 and data pool 3 2024-04-22T06:35:40.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.805+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 msgr2=0x7f54180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.805+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 0x7f54180a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5430199760 tx=0x7f542400a560 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 msgr2=0x7f5430198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f542000b4a0 tx=0x7f542000b960 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 shutdown_connections 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f54180a3c30 0x7f54180a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54301029a0 0x7f543019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5430101fe0 0x7f5430198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 --2- 172.21.15.105:0/3808596932 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5430100e90 0x7f5430198250 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 >> 172.21.15.105:0/3808596932 conn(0x7f54300fc400 msgr2=0x7f54301053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 shutdown_connections 2024-04-22T06:35:40.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.806+0000 7f543666d640 1 -- 172.21.15.105:0/3808596932 wait complete. 2024-04-22T06:35:40.827 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:35:40.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 -- 172.21.15.105:0/763836173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 msgr2=0x7f9c601093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 --2- 172.21.15.105:0/763836173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c601093b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009ca0 tx=0x7f9c5802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 -- 172.21.15.105:0/763836173 shutdown_connections 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 --2- 172.21.15.105:0/763836173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c601093b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 --2- 172.21.15.105:0/763836173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60101150 0x7f9c601015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 --2- 172.21.15.105:0/763836173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60102420 0x7f9c60102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 -- 172.21.15.105:0/763836173 >> 172.21.15.105:0/763836173 conn(0x7f9c600fbb80 msgr2=0x7f9c600fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 -- 172.21.15.105:0/763836173 shutdown_connections 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.929+0000 7f9c64e21640 1 -- 172.21.15.105:0/763836173 wait complete. 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 Processor -- start 2024-04-22T06:35:40.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 -- start start 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60101150 0x7f9c60104e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60102420 0x7f9c60103a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6010c9c0 con 0x7f9c60101b10 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6010c840 con 0x7f9c60102420 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c64e21640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6010cb40 con 0x7f9c60101150 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48606/0 (socket says 172.21.15.105:48606) 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 -- 172.21.15.105:0/2775037637 learned_addr learned my addr 172.21.15.105:0/2775037637 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c5e59c640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60101150 0x7f9c60104e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c5ed9d640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60102420 0x7f9c60103a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 -- 172.21.15.105:0/2775037637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60101150 msgr2=0x7f9c60104e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60101150 0x7f9c60104e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 -- 172.21.15.105:0/2775037637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60102420 msgr2=0x7f9c60103a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:40.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60102420 0x7f9c60103a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.930+0000 7f9c57fff640 1 -- 172.21.15.105:0/2775037637 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c601042a0 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c57fff640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800c890 tx=0x7f9c4800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c48004220 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c480043c0 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c601a2280 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c48004a00 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.931+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c601a27d0 con 0x7f9c60101b10 2024-04-22T06:35:40.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.932+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c601a2a90 con 0x7f9c60101b10 2024-04-22T06:35:40.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.933+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9c480079b0 con 0x7f9c60101b10 2024-04-22T06:35:40.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.933+0000 7f9c577fe640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 0x7f9c4c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:40.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.933+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7f9c480cfff0 con 0x7f9c60101b10 2024-04-22T06:35:40.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.933+0000 7f9c5e59c640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 0x7f9c4c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:40.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.934+0000 7f9c5e59c640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 0x7f9c4c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9c50005fd0 tx=0x7f9c5000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:40.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:40.938+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c48014030 con 0x7f9c60101b10 2024-04-22T06:35:41.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.118+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c601a2c60 con 0x7f9c60101b10 2024-04-22T06:35:41.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.120+0000 7f9c577fe640 1 -- 172.21.15.105:0/2775037637 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22278 (secure 0 0 0) 0x7f9c4809cb20 con 0x7f9c60101b10 2024-04-22T06:35:41.119 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:41.119 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":88,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:35:40.134371+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 msgr2=0x7f9c4c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 0x7f9c4c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9c50005fd0 tx=0x7f9c5000b390 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 msgr2=0x7f9c60103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800c890 tx=0x7f9c4800cd50 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 shutdown_connections 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f9c4c0a3bb0 0x7f9c4c0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c60102420 0x7f9c60103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c60101b10 0x7f9c60103540 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 --2- 172.21.15.105:0/2775037637 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9c60101150 0x7f9c60104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 >> 172.21.15.105:0/2775037637 conn(0x7f9c600fbb80 msgr2=0x7f9c600fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 shutdown_connections 2024-04-22T06:35:41.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.122+0000 7f9c64e21640 1 -- 172.21.15.105:0/2775037637 wait complete. 2024-04-22T06:35:41.130 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:35:41.130 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:40 smithi105 ceph-mon[27916]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:35:41.130 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:35:41.130 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:40 smithi105 ceph-mon[27916]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:35:41.130 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:40 smithi105 ceph-mon[27916]: pgmap v285: 97 pgs: 14 creating+peering, 10 active+clean, 73 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:41.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:35:41.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:35:41.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:40 smithi112 ceph-mon[31236]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:35:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:35:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:40 smithi112 ceph-mon[31236]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:35:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:40 smithi112 ceph-mon[31236]: pgmap v285: 97 pgs: 14 creating+peering, 10 active+clean, 73 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:41.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 -- 172.21.15.105:0/2635435171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 msgr2=0x7ff11c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 --2- 172.21.15.105:0/2635435171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c1092b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff114009ca0 tx=0x7ff11402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 -- 172.21.15.105:0/2635435171 shutdown_connections 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 --2- 172.21.15.105:0/2635435171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c1092b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 --2- 172.21.15.105:0/2635435171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c101fd0 0x7ff11c102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 --2- 172.21.15.105:0/2635435171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c100e80 0x7ff11c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 -- 172.21.15.105:0/2635435171 >> 172.21.15.105:0/2635435171 conn(0x7ff11c0fc410 msgr2=0x7ff11c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 -- 172.21.15.105:0/2635435171 shutdown_connections 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.231+0000 7ff120d96640 1 -- 172.21.15.105:0/2635435171 wait complete. 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 Processor -- start 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 -- start start 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c100e80 0x7ff11c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c101fd0 0x7ff11c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10db20 con 0x7ff11c102990 2024-04-22T06:35:41.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10d9a0 con 0x7ff11c100e80 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c10dca0 con 0x7ff11c101fd0 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48634/0 (socket says 172.21.15.105:48634) 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 -- 172.21.15.105:0/4142717109 learned_addr learned my addr 172.21.15.105:0/4142717109 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff119d9b640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c101fd0 0x7ff11c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11a59c640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c100e80 0x7ff11c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 -- 172.21.15.105:0/4142717109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c101fd0 msgr2=0x7ff11c198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c101fd0 0x7ff11c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 -- 172.21.15.105:0/4142717109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c100e80 msgr2=0x7ff11c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c100e80 0x7ff11c1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 -- 172.21.15.105:0/4142717109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c19fce0 con 0x7ff11c102990 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff11ad9d640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff11402c700 tx=0x7ff114034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114039040 con 0x7ff11c102990 2024-04-22T06:35:41.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.232+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c19ff70 con 0x7ff11c102990 2024-04-22T06:35:41.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.233+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff114044070 con 0x7ff11c102990 2024-04-22T06:35:41.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.233+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1140042a0 con 0x7ff11c102990 2024-04-22T06:35:41.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.233+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1a03e0 con 0x7ff11c102990 2024-04-22T06:35:41.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.233+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c108190 con 0x7ff11c102990 2024-04-22T06:35:41.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.235+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff114003970 con 0x7ff11c102990 2024-04-22T06:35:41.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.235+0000 7ff1137fe640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 0x7ff1080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.235+0000 7ff11a59c640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 0x7ff1080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.235+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ff1140f31c0 con 0x7ff11c102990 2024-04-22T06:35:41.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.236+0000 7ff11a59c640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 0x7ff1080a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff104005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.239+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1140bfcf0 con 0x7ff11c102990 2024-04-22T06:35:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2821531077' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-22T06:35:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:40 smithi177 ceph-mon[31346]: osdmap e87: 12 total, 12 up, 12 in 2024-04-22T06:35:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-22T06:35:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:40 smithi177 ceph-mon[31346]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-22T06:35:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:40 smithi177 ceph-mon[31346]: pgmap v285: 97 pgs: 14 creating+peering, 10 active+clean, 73 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:35:41.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.418+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff11c02d810 con 0x7ff11c102990 2024-04-22T06:35:41.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.419+0000 7ff1137fe640 1 -- 172.21.15.105:0/4142717109 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22278 (secure 0 0 0) 0x7ff114051e40 con 0x7ff11c102990 2024-04-22T06:35:41.418 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:41.418 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":88,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:35:40.134371+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 msgr2=0x7ff1080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 0x7ff1080a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff104005850 comp rx=0 tx=0).stop 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 msgr2=0x7ff11c19f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff11402c700 tx=0x7ff114034760 comp rx=0 tx=0).stop 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 shutdown_connections 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff1080a3bb0 0x7ff1080a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff11c102990 0x7ff11c19f670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff11c101fd0 0x7ff11c198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 --2- 172.21.15.105:0/4142717109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff11c100e80 0x7ff11c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 >> 172.21.15.105:0/4142717109 conn(0x7ff11c0fc410 msgr2=0x7ff11c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 shutdown_connections 2024-04-22T06:35:41.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.421+0000 7ff120d96640 1 -- 172.21.15.105:0/4142717109 wait complete. 2024-04-22T06:35:41.431 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- 172.21.15.105:0/2033210039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc105330 msgr2=0x7ffacc107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 --2- 172.21.15.105:0/2033210039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc105330 0x7ffacc107720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009900 tx=0x7ffabc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- 172.21.15.105:0/2033210039 shutdown_connections 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 --2- 172.21.15.105:0/2033210039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc107c60 0x7ffacc10a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 --2- 172.21.15.105:0/2033210039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc105330 0x7ffacc107720 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 --2- 172.21.15.105:0/2033210039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc102a00 0x7ffacc104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- 172.21.15.105:0/2033210039 >> 172.21.15.105:0/2033210039 conn(0x7ffacc0fc3f0 msgr2=0x7ffacc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- 172.21.15.105:0/2033210039 shutdown_connections 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- 172.21.15.105:0/2033210039 wait complete. 2024-04-22T06:35:41.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 Processor -- start 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.534+0000 7ffad0f15640 1 -- start start 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 0x7ffacc198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc107c60 0x7ffacc19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10db80 con 0x7ffacc102a00 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10da00 con 0x7ffacc107c60 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10dd00 con 0x7ffacc105330 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48652/0 (socket says 172.21.15.105:48652) 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 -- 172.21.15.105:0/686193658 learned_addr learned my addr 172.21.15.105:0/686193658 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffac9d9b640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 0x7ffacc198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffacad9d640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc107c60 0x7ffacc19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 -- 172.21.15.105:0/686193658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 msgr2=0x7ffacc198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 0x7ffacc198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 -- 172.21.15.105:0/686193658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc107c60 msgr2=0x7ffacc19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc107c60 0x7ffacc19f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 -- 172.21.15.105:0/686193658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc19ff60 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffac9d9b640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 0x7ffacc198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffaca59c640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffac000b4a0 tx=0x7ffac000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0014070 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac0010ac0 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc1a0250 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0017c30 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.535+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1a07f0 con 0x7ffacc102a00 2024-04-22T06:35:41.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.536+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc0fdee0 con 0x7ffacc102a00 2024-04-22T06:35:41.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.537+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ffac0017070 con 0x7ffacc102a00 2024-04-22T06:35:41.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.538+0000 7ffabb7fe640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 0x7ffab40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:41.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.538+0000 7ffac9d9b640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 0x7ffab40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:41.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.538+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7979+0+0 (secure 0 0 0) 0x7ffac00cfa40 con 0x7ffacc102a00 2024-04-22T06:35:41.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.538+0000 7ffac9d9b640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 0x7ffab40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006010 tx=0x7ffabc0095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:41.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.543+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffac009c570 con 0x7ffacc102a00 2024-04-22T06:35:41.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.760+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ffacc02d5c0 con 0x7ffacc102a00 2024-04-22T06:35:41.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:41.999+0000 7ffabb7fe640 1 -- 172.21.15.105:0/686193658 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7ffac009be70 con 0x7ffacc102a00 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 msgr2=0x7ffab40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 0x7ffab40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006010 tx=0x7ffabc0095f0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 msgr2=0x7ffacc198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffac000b4a0 tx=0x7ffac000b960 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 shutdown_connections 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ffab40a3bb0 0x7ffab40a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffacc107c60 0x7ffacc19f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffacc105330 0x7ffacc198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 --2- 172.21.15.105:0/686193658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffacc102a00 0x7ffacc198340 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 >> 172.21.15.105:0/686193658 conn(0x7ffacc0fc3f0 msgr2=0x7ffacc104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:42.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 shutdown_connections 2024-04-22T06:35:42.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.001+0000 7ffad0f15640 1 -- 172.21.15.105:0/686193658 wait complete. 2024-04-22T06:35:42.011 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-22T06:35:42.011 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-22T06:35:42.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 -- 172.21.15.105:0/2097645515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834107c70 msgr2=0x7f983410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:42.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 --2- 172.21.15.105:0/2097645515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834107c70 0x7f983410a0c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f982c009ca0 tx=0x7f982c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 -- 172.21.15.105:0/2097645515 shutdown_connections 2024-04-22T06:35:42.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 --2- 172.21.15.105:0/2097645515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834107c70 0x7f983410a0c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 --2- 172.21.15.105:0/2097645515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 --2- 172.21.15.105:0/2097645515 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9834102a10 0x7f9834104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 -- 172.21.15.105:0/2097645515 >> 172.21.15.105:0/2097645515 conn(0x7f98340fc400 msgr2=0x7f98340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 -- 172.21.15.105:0/2097645515 shutdown_connections 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.115+0000 7f983997e640 1 -- 172.21.15.105:0/2097645515 wait complete. 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 Processor -- start 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 -- start start 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9834107c70 0x7f983419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f983410db40 con 0x7f9834102a10 2024-04-22T06:35:42.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f983410d9c0 con 0x7f9834105340 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f983997e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f983410dcc0 con 0x7f9834107c70 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48664/0 (socket says 172.21.15.105:48664) 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 -- 172.21.15.105:0/2373116351 learned_addr learned my addr 172.21.15.105:0/2373116351 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f98327fc640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 -- 172.21.15.105:0/2373116351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9834107c70 msgr2=0x7f983419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9834107c70 0x7f983419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 -- 172.21.15.105:0/2373116351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 msgr2=0x7f9834198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f9832ffd640 1 -- 172.21.15.105:0/2373116351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983419fe60 con 0x7f9834102a10 2024-04-22T06:35:42.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.116+0000 7f98327fc640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f9832ffd640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f982400b4a0 tx=0x7f982400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824014070 con 0x7f9834102a10 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9824010ac0 con 0x7f9834102a10 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98341a0150 con 0x7f9834102a10 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824017c30 con 0x7f9834102a10 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.117+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98341a0650 con 0x7f9834102a10 2024-04-22T06:35:42.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.118+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98340fdef0 con 0x7f9834102a10 2024-04-22T06:35:42.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.119+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9824017070 con 0x7f9834102a10 2024-04-22T06:35:42.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.119+0000 7f983897c640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:42.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.119+0000 7f98327fc640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:42.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.119+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f98240cf700 con 0x7f9834102a10 2024-04-22T06:35:42.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.120+0000 7f98327fc640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 0x7f981c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f98280058c0 tx=0x7f9828005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:42.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.124+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f982409c230 con 0x7f9834102a10 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:35:42.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:active 2024-04-22T06:35:42.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:42.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2775037637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4142717109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:41 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:35:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:active 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2775037637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4142717109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:41 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: osdmap e88: 12 total, 12 up, 12 in 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3808596932' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] up:active 2024-04-22T06:35:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2775037637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4142717109' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:41 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-22T06:35:42.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:42.339+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f9834062b50 con 0x7f9834102a10 2024-04-22T06:35:43.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.007+0000 7f983897c640 1 -- 172.21.15.105:0/2373116351 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7f982409bb30 con 0x7f9834102a10 2024-04-22T06:35:43.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.009+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 msgr2=0x7f981c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 0x7f981c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f98280058c0 tx=0x7f9828005850 comp rx=0 tx=0).stop 2024-04-22T06:35:43.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 msgr2=0x7f9834198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f982400b4a0 tx=0x7f982400b960 comp rx=0 tx=0).stop 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 shutdown_connections 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f981c0a3bb0 0x7f981c0a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9834107c70 0x7f983419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9834105340 0x7f9834198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 --2- 172.21.15.105:0/2373116351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9834102a10 0x7f9834198240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 >> 172.21.15.105:0/2373116351 conn(0x7f98340fc400 msgr2=0x7f98341032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 shutdown_connections 2024-04-22T06:35:43.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.010+0000 7f983997e640 1 -- 172.21.15.105:0/2373116351 wait complete. 2024-04-22T06:35:43.023 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/768524131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 msgr2=0x7fb2e81012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/768524131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81012a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009900 tx=0x7fb2d802bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/768524131 shutdown_connections 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/768524131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 0x7fb2e81092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/768524131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 0x7fb2e8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/768524131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81012a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/768524131 >> 172.21.15.105:0/768524131 conn(0x7fb2e80fc3f0 msgr2=0x7fb2e80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/768524131 shutdown_connections 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/768524131 wait complete. 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 Processor -- start 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.127+0000 7fb2ef3fd640 1 -- start start 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 0x7fb2e81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 0x7fb2e819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810db40 con 0x7fb2e8100e80 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d9c0 con 0x7fb2e8102990 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810dcc0 con 0x7fb2e8101fd0 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48682/0 (socket says 172.21.15.105:48682) 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 -- 172.21.15.105:0/2344888008 learned_addr learned my addr 172.21.15.105:0/2344888008 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:43.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ec998640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 0x7fb2e81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed99a640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 0x7fb2e819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 -- 172.21.15.105:0/2344888008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 msgr2=0x7fb2e81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 0x7fb2e81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 -- 172.21.15.105:0/2344888008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 msgr2=0x7fb2e819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 0x7fb2e819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 -- 172.21.15.105:0/2344888008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e819fdc0 con 0x7fb2e8100e80 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ed199640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8000c00 tx=0x7fb2d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8043020 con 0x7fb2e8100e80 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d80054c0 con 0x7fb2e8100e80 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e81a0050 con 0x7fb2e8100e80 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8003d80 con 0x7fb2e8100e80 2024-04-22T06:35:43.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.128+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e81a05a0 con 0x7fb2e8100e80 2024-04-22T06:35:43.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.129+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e8108190 con 0x7fb2e8100e80 2024-04-22T06:35:43.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.130+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb2d8046020 con 0x7fb2e8100e80 2024-04-22T06:35:43.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.131+0000 7fb2de7fc640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 0x7fb2d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:43.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.131+0000 7fb2ec998640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 0x7fb2d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:43.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.131+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fb2d80f2640 con 0x7fb2e8100e80 2024-04-22T06:35:43.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.131+0000 7fb2ec998640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 0x7fb2d40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb2e81996b0 tx=0x7fb2e400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:43.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.135+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2d804d020 con 0x7fb2e8100e80 2024-04-22T06:35:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:42 smithi105 ceph-mon[27916]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:35:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:35:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:42 smithi105 ceph-mon[27916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T06:35:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:42 smithi105 ceph-mon[27916]: pgmap v288: 97 pgs: 14 creating+peering, 41 active+clean, 42 unknown; 578 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-04-22T06:35:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:42 smithi112 ceph-mon[31236]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:35:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:35:43.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:42 smithi112 ceph-mon[31236]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:43.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T06:35:43.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:42 smithi112 ceph-mon[31236]: pgmap v288: 97 pgs: 14 creating+peering, 41 active+clean, 42 unknown; 578 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-04-22T06:35:43.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:42 smithi177 ceph-mon[31346]: osdmap e89: 12 total, 12 up, 12 in 2024-04-22T06:35:43.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/686193658' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-22T06:35:43.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:42 smithi177 ceph-mon[31346]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:43.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-22T06:35:43.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:42 smithi177 ceph-mon[31346]: pgmap v288: 97 pgs: 14 creating+peering, 41 active+clean, 42 unknown; 578 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2024-04-22T06:35:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:43.353+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb2e8062b50 con 0x7fb2e8100e80 2024-04-22T06:35:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.016+0000 7fb2de7fc640 1 -- 172.21.15.105:0/2344888008 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7fb2d80bf290 con 0x7fb2e8100e80 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 msgr2=0x7fb2d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 0x7fb2d40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb2e81996b0 tx=0x7fb2e400a2d0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 msgr2=0x7fb2e81981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8000c00 tx=0x7fb2d80042f0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 shutdown_connections 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb2d40a3bb0 0x7fb2d40a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2e8102990 0x7fb2e819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb2e8101fd0 0x7fb2e81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 --2- 172.21.15.105:0/2344888008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb2e8100e80 0x7fb2e81981a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 >> 172.21.15.105:0/2344888008 conn(0x7fb2e80fc3f0 msgr2=0x7fb2e8105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:44.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 shutdown_connections 2024-04-22T06:35:44.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.019+0000 7fb2ef3fd640 1 -- 172.21.15.105:0/2344888008 wait complete. 2024-04-22T06:35:44.033 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-22T06:35:44.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 -- 172.21.15.105:0/1079149534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 msgr2=0x7faa4410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 --2- 172.21.15.105:0/1079149534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4410a0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faa34009900 tx=0x7faa3402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 -- 172.21.15.105:0/1079149534 shutdown_connections 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 --2- 172.21.15.105:0/1079149534 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4410a0e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 --2- 172.21.15.105:0/1079149534 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44105360 0x7faa44107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 --2- 172.21.15.105:0/1079149534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44102a30 0x7faa44104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 -- 172.21.15.105:0/1079149534 >> 172.21.15.105:0/1079149534 conn(0x7faa440fc440 msgr2=0x7faa440fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 -- 172.21.15.105:0/1079149534 shutdown_connections 2024-04-22T06:35:44.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.138+0000 7faa48a25640 1 -- 172.21.15.105:0/1079149534 wait complete. 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 Processor -- start 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 -- start start 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 0x7faa44198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44105360 0x7faa44198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410db50 con 0x7faa44107c90 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410d9d0 con 0x7faa44105360 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410dcd0 con 0x7faa44102a30 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48704/0 (socket says 172.21.15.105:48704) 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 -- 172.21.15.105:0/2245092610 learned_addr learned my addr 172.21.15.105:0/2245092610 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa41d9b640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44105360 0x7faa44198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:44.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa4259c640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 0x7faa44198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 -- 172.21.15.105:0/2245092610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 msgr2=0x7faa44198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 0x7faa44198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 -- 172.21.15.105:0/2245092610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44105360 msgr2=0x7faa44198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44105360 0x7faa44198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 -- 172.21.15.105:0/2245092610 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4419fe40 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa4259c640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 0x7faa44198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa42d9d640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faa34009230 tx=0x7faa340040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34043020 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa441a00d0 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa340044c0 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.139+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34003bc0 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.140+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa441a06a0 con 0x7faa44107c90 2024-04-22T06:35:44.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.140+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa440fdf30 con 0x7faa44107c90 2024-04-22T06:35:44.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.142+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7faa34046020 con 0x7faa44107c90 2024-04-22T06:35:44.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.143+0000 7faa337fe640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 0x7faa2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:44.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.143+0000 7faa4259c640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 0x7faa2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:44.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.143+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7faa340f29d0 con 0x7faa44107c90 2024-04-22T06:35:44.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.143+0000 7faa4259c640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 0x7faa2c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:44.145 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.147+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa3404d020 con 0x7faa44107c90 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T06:35:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:43 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:35:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:44.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T06:35:44.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T06:35:44.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:43 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2373116351' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-22T06:35:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:43 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-22T06:35:44.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:44.363+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7faa4402d5c0 con 0x7faa44107c90 2024-04-22T06:35:45.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.027+0000 7faa337fe640 1 -- 172.21.15.105:0/2245092610 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7faa340bf500 con 0x7faa44107c90 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 msgr2=0x7faa2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 0x7faa2c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009c20 tx=0x7faa3c005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 msgr2=0x7faa4419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faa34009230 tx=0x7faa340040c0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 shutdown_connections 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7faa2c0a3bb0 0x7faa2c0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44107c90 0x7faa4419f7d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa44105360 0x7faa44198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 --2- 172.21.15.105:0/2245092610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7faa44102a30 0x7faa44198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 >> 172.21.15.105:0/2245092610 conn(0x7faa440fc440 msgr2=0x7faa44103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.030+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 shutdown_connections 2024-04-22T06:35:45.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.031+0000 7faa48a25640 1 -- 172.21.15.105:0/2245092610 wait complete. 2024-04-22T06:35:45.044 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-22T06:35:45.044 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 -- 172.21.15.105:0/1647928034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630101fa0 msgr2=0x7f86301023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 --2- 172.21.15.105:0/1647928034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630101fa0 0x7f86301023c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f862c009900 tx=0x7f862c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 -- 172.21.15.105:0/1647928034 shutdown_connections 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 --2- 172.21.15.105:0/1647928034 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630103b50 0x7f863010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 --2- 172.21.15.105:0/1647928034 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103190 0x7f8630103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 --2- 172.21.15.105:0/1647928034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630101fa0 0x7f86301023c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 -- 172.21.15.105:0/1647928034 >> 172.21.15.105:0/1647928034 conn(0x7f86300fd530 msgr2=0x7f86300ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 -- 172.21.15.105:0/1647928034 shutdown_connections 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 -- 172.21.15.105:0/1647928034 wait complete. 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.147+0000 7f8637b96640 1 Processor -- start 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- start start 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 0x7f8630198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103b50 0x7f863019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f863010db30 con 0x7f8630103190 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f863010d9b0 con 0x7f8630101fa0 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f863010dcb0 con 0x7f8630103b50 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49832/0 (socket says 172.21.15.105:49832) 2024-04-22T06:35:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 -- 172.21.15.105:0/1244059464 learned_addr learned my addr 172.21.15.105:0/1244059464 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635932640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 0x7f8630198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8636133640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103b50 0x7f863019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 -- 172.21.15.105:0/1244059464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103b50 msgr2=0x7f863019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103b50 0x7f863019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 -- 172.21.15.105:0/1244059464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 msgr2=0x7f8630198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 0x7f8630198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 -- 172.21.15.105:0/1244059464 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863019fd90 con 0x7f8630103190 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635932640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 0x7f8630198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8635131640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f862000c920 tx=0x7f862000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007c70 con 0x7f8630103190 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86301a0080 con 0x7f8630103190 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8620007e10 con 0x7f8630103190 2024-04-22T06:35:45.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620018820 con 0x7f8630103190 2024-04-22T06:35:45.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.148+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86301a0620 con 0x7f8630103190 2024-04-22T06:35:45.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.149+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86301092c0 con 0x7f8630103190 2024-04-22T06:35:45.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.150+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f862000f450 con 0x7f8630103190 2024-04-22T06:35:45.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.151+0000 7f8626ffd640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 0x7f861c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.151+0000 7f8635932640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 0x7f861c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.151+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f86200d0310 con 0x7f8630103190 2024-04-22T06:35:45.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.151+0000 7f8635932640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 0x7f861c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f862c02c240 tx=0x7f862c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.155+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8620014030 con 0x7f8630103190 2024-04-22T06:35:45.355 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.356+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8630062390 con 0x7f861c0a3bb0 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:active 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-22T06:35:45.360 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-22T06:35:45.361 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:35:45.361 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-mon[27916]: pgmap v289: 97 pgs: 12 creating+peering, 85 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-04-22T06:35:45.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.379+0000 7f8626ffd640 1 -- 172.21.15.105:0/1244059464 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8630062390 con 0x7f861c0a3bb0 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 msgr2=0x7f861c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 0x7f861c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f862c02c240 tx=0x7f862c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 msgr2=0x7f86301986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f862000c920 tx=0x7f862000cde0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 shutdown_connections 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f861c0a3bb0 0x7f861c0a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8630103b50 0x7f863019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8630103190 0x7f86301986b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 --2- 172.21.15.105:0/1244059464 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8630101fa0 0x7f8630198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 >> 172.21.15.105:0/1244059464 conn(0x7f86300fd530 msgr2=0x7f8630106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 shutdown_connections 2024-04-22T06:35:45.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.381+0000 7f8637b96640 1 -- 172.21.15.105:0/1244059464 wait complete. 2024-04-22T06:35:45.391 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-22T06:35:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:35:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:active 2024-04-22T06:35:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:35:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T06:35:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-22T06:35:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-22T06:35:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:35:45.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-mon[31236]: pgmap v289: 97 pgs: 12 creating+peering, 85 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2344888008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] up:active 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-22T06:35:45.492 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:45 smithi177 ceph-mon[31346]: pgmap v289: 97 pgs: 12 creating+peering, 85 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-04-22T06:35:45.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- 172.21.15.105:0/3149045439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 msgr2=0x7ff864074380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 --2- 172.21.15.105:0/3149045439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff864074380 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff854009900 tx=0x7ff85402bcd0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- 172.21.15.105:0/3149045439 shutdown_connections 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 --2- 172.21.15.105:0/3149045439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff86410c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 --2- 172.21.15.105:0/3149045439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 0x7ff864074db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 --2- 172.21.15.105:0/3149045439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff864074380 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- 172.21.15.105:0/3149045439 >> 172.21.15.105:0/3149045439 conn(0x7ff8640fd530 msgr2=0x7ff8640ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- 172.21.15.105:0/3149045439 shutdown_connections 2024-04-22T06:35:45.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- 172.21.15.105:0/3149045439 wait complete. 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 Processor -- start 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.496+0000 7ff86a47a640 1 -- start start 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 0x7ff86419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff8641a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410fce0 con 0x7ff864075f30 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410fb60 con 0x7ff864109e30 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff86410fe60 con 0x7ff864074950 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49868/0 (socket says 172.21.15.105:49868) 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 -- 172.21.15.105:0/495502458 learned_addr learned my addr 172.21.15.105:0/495502458 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff863fff640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 0x7ff86419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff868a17640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff8641a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 -- 172.21.15.105:0/495502458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 msgr2=0x7ff86419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 0x7ff86419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 -- 172.21.15.105:0/495502458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 msgr2=0x7ff8641a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff8641a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 -- 172.21.15.105:0/495502458 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8641a2060 con 0x7ff864075f30 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff868a17640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff8641a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:35:45.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8637fe640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff85800e9f0 tx=0x7ff85800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85800cd80 con 0x7ff864075f30 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8641a2350 con 0x7ff864075f30 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8580054d0 con 0x7ff864075f30 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858010690 con 0x7ff864075f30 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.497+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8641a28a0 con 0x7ff864075f30 2024-04-22T06:35:45.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.499+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff864108bf0 con 0x7ff864075f30 2024-04-22T06:35:45.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.500+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff858004e90 con 0x7ff864075f30 2024-04-22T06:35:45.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.500+0000 7ff8617fa640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 0x7ff84c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.500+0000 7ff863fff640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 0x7ff84c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.500+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7ff8580d0750 con 0x7ff864075f30 2024-04-22T06:35:45.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.501+0000 7ff863fff640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 0x7ff84c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff854009230 tx=0x7ff854005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.504+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff85809d280 con 0x7ff864075f30 2024-04-22T06:35:45.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:35:45.361+0000 7fda58b35700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:35:45.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:35:45.361+0000 7fda58b35700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:35:45.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:35:45.361+0000 7fda58b35700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:35:45.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:35:45.361+0000 7fda58b35700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:35:45.668 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:35:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:35:45.361+0000 7fda58b35700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:35:45.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.712+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff864062390 con 0x7ff84c0a3bb0 2024-04-22T06:35:45.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.736+0000 7ff8617fa640 1 -- 172.21.15.105:0/495502458 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff864062390 con 0x7ff84c0a3bb0 2024-04-22T06:35:45.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.737+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 msgr2=0x7ff84c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.737+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 0x7ff84c0a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff854009230 tx=0x7ff854005fd0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.737+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 msgr2=0x7ff86419a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.737+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff85800e9f0 tx=0x7ff85800edf0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.736 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 shutdown_connections 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff84c0a3bb0 0x7ff84c0a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff864109e30 0x7ff8641a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff864075f30 0x7ff86419a980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 --2- 172.21.15.105:0/495502458 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff864074950 0x7ff86419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 >> 172.21.15.105:0/495502458 conn(0x7ff8640fd530 msgr2=0x7ff864105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 shutdown_connections 2024-04-22T06:35:45.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.738+0000 7ff86a47a640 1 -- 172.21.15.105:0/495502458 wait complete. 2024-04-22T06:35:45.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-22T06:35:45.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 -- 172.21.15.105:0/3183919858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103170 msgr2=0x7fc22c1035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 --2- 172.21.15.105:0/3183919858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103170 0x7fc22c1035f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc220009900 tx=0x7fc22002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:45.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 -- 172.21.15.105:0/3183919858 shutdown_connections 2024-04-22T06:35:45.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 --2- 172.21.15.105:0/3183919858 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103b30 0x7fc22c10a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 --2- 172.21.15.105:0/3183919858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103170 0x7fc22c1035f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 --2- 172.21.15.105:0/3183919858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 -- 172.21.15.105:0/3183919858 >> 172.21.15.105:0/3183919858 conn(0x7fc22c0fd510 msgr2=0x7fc22c0ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 -- 172.21.15.105:0/3183919858 shutdown_connections 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.852+0000 7fc2328e0640 1 -- 172.21.15.105:0/3183919858 wait complete. 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 Processor -- start 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 -- start start 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103170 0x7fc22c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10dac0 con 0x7fc22c103b30 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10d940 con 0x7fc22c101f80 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc2328e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10dc40 con 0x7fc22c103170 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49878/0 (socket says 172.21.15.105:49878) 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 -- 172.21.15.105:0/1317744315 learned_addr learned my addr 172.21.15.105:0/1317744315 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc22bfff640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc22b7fe640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103170 0x7fc22c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 -- 172.21.15.105:0/1317744315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103170 msgr2=0x7fc22c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103170 0x7fc22c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 -- 172.21.15.105:0/1317744315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 msgr2=0x7fc22c198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 -- 172.21.15.105:0/1317744315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c19fda0 con 0x7fc22c103b30 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc22bfff640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:45.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.853+0000 7fc230e7d640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc22400eaa0 tx=0x7fc22400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.854+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22400cdb0 con 0x7fc22c103b30 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.854+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc224005020 con 0x7fc22c103b30 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.854+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c1a0090 con 0x7fc22c103b30 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.854+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240116d0 con 0x7fc22c103b30 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.854+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c1a05e0 con 0x7fc22c103b30 2024-04-22T06:35:45.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.855+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc22c109310 con 0x7fc22c103b30 2024-04-22T06:35:45.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.856+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc2240051d0 con 0x7fc22c103b30 2024-04-22T06:35:45.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.856+0000 7fc2297fa640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:45.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.856+0000 7fc22bfff640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:45.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.856+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fc2240d07d0 con 0x7fc22c103b30 2024-04-22T06:35:45.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.857+0000 7fc22bfff640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 0x7fc2140a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc22c102fd0 tx=0x7fc21c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:45.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:45.860+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc224016030 con 0x7fc22c103b30 2024-04-22T06:35:45.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:35:45.676+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-22T06:35:45.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:35:45.676+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:35:45.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:35:45.676+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:35:46.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.076+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fc22c062390 con 0x7fc2140a3c30 2024-04-22T06:35:46.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.081+0000 7fc2297fa640 1 -- 172.21.15.105:0/1317744315 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc22c062390 con 0x7fc2140a3c30 2024-04-22T06:35:46.079 INFO:teuthology.orchestra.run.smithi105.stdout:/volumes/qa/sv_0/2acd25bd-4fa8-4535-a580-a391aa6fbb8b 2024-04-22T06:35:46.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 msgr2=0x7fc2140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 0x7fc2140a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc22c102fd0 tx=0x7fc21c005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 msgr2=0x7fc22c19f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc22400eaa0 tx=0x7fc22400ef60 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 shutdown_connections 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc22c103b30 0x7fc22c19f730 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc22c103170 0x7fc22c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 --2- 172.21.15.105:0/1317744315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc22c101f80 0x7fc22c198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 >> 172.21.15.105:0/1317744315 conn(0x7fc22c0fd510 msgr2=0x7fc22c1064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 shutdown_connections 2024-04-22T06:35:46.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.083+0000 7fc2328e0640 1 -- 172.21.15.105:0/1317744315 wait complete. 2024-04-22T06:35:46.092 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-22T06:35:46.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.196+0000 7f53d4b52640 1 -- 172.21.15.105:0/3639865525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0107cb0 msgr2=0x7f53d010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.196+0000 7f53d4b52640 1 --2- 172.21.15.105:0/3639865525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0107cb0 0x7f53d010a0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009ca0 tx=0x7f53c802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- 172.21.15.105:0/3639865525 shutdown_connections 2024-04-22T06:35:46.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- 172.21.15.105:0/3639865525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0107cb0 0x7f53d010a0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- 172.21.15.105:0/3639865525 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53d0105380 0x7f53d0107770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- 172.21.15.105:0/3639865525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0102a50 0x7f53d0104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- 172.21.15.105:0/3639865525 >> 172.21.15.105:0/3639865525 conn(0x7f53d00fc400 msgr2=0x7f53d00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- 172.21.15.105:0/3639865525 shutdown_connections 2024-04-22T06:35:46.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- 172.21.15.105:0/3639865525 wait complete. 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 Processor -- start 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- start start 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53d0105380 0x7f53d0100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0107cb0 0x7f53d0100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53d010dac0 con 0x7f53d0102a50 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53d010d940 con 0x7f53d0107cb0 2024-04-22T06:35:46.198 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.197+0000 7f53d4b52640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53d010dc40 con 0x7f53d0105380 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49908/0 (socket says 172.21.15.105:49908) 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 -- 172.21.15.105:0/2822945431 learned_addr learned my addr 172.21.15.105:0/2822945431 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ced9d640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0107cb0 0x7f53d0100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 -- 172.21.15.105:0/2822945431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53d0105380 msgr2=0x7f53d0100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53d0105380 0x7f53d0100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 -- 172.21.15.105:0/2822945431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0107cb0 msgr2=0x7f53d0100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0107cb0 0x7f53d0100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 -- 172.21.15.105:0/2822945431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d0101800 con 0x7f53d0102a50 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53ce59c640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e9f0 tx=0x7f53c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c000cd80 con 0x7f53d0102a50 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d01a2360 con 0x7f53d0102a50 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c00108e0 con 0x7f53d0102a50 2024-04-22T06:35:46.199 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0018980 con 0x7f53d0102a50 2024-04-22T06:35:46.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.198+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d01a2880 con 0x7f53d0102a50 2024-04-22T06:35:46.200 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.199+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d00fdef0 con 0x7f53d0102a50 2024-04-22T06:35:46.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.200+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f53c0010430 con 0x7f53d0102a50 2024-04-22T06:35:46.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.201+0000 7f53bf7fe640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 0x7f53b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.201+0000 7f53cdd9b640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 0x7f53b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.201+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f53c00d0610 con 0x7f53d0102a50 2024-04-22T06:35:46.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.201+0000 7f53cdd9b640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 0x7f53b80a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f53c4005fd0 tx=0x7f53c4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.205+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53c009d140 con 0x7f53d0102a50 2024-04-22T06:35:46.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.411+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f53d0062420 con 0x7f53b80a3bb0 2024-04-22T06:35:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:35:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:active 2024-04-22T06:35:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T06:35:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:active 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-22T06:35:46.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:46 smithi105 ceph-mon[27916]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:35:46.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.434+0000 7f53bf7fe640 1 -- 172.21.15.105:0/2822945431 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f53d0062420 con 0x7f53b80a3bb0 2024-04-22T06:35:46.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 msgr2=0x7f53b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 0x7f53b80a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f53c4005fd0 tx=0x7f53c4005850 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 msgr2=0x7f53d0102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e9f0 tx=0x7f53c000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 shutdown_connections 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53b80a3bb0 0x7f53b80a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53d0107cb0 0x7f53d0100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53d0105380 0x7f53d0100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 --2- 172.21.15.105:0/2822945431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53d0102a50 0x7f53d0102370 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 >> 172.21.15.105:0/2822945431 conn(0x7f53d00fc400 msgr2=0x7f53d0103390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 shutdown_connections 2024-04-22T06:35:46.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.437+0000 7f53d4b52640 1 -- 172.21.15.105:0/2822945431 wait complete. 2024-04-22T06:35:46.446 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-22T06:35:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:35:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:active 2024-04-22T06:35:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:active 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-22T06:35:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:46 smithi112 ceph-mon[31236]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2245092610' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] up:active 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] up:active 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:35:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-04-22T06:35:46.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-22T06:35:46.491 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:46 smithi177 ceph-mon[31346]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:35:46.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/1244101224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103170 msgr2=0x7fa9c41035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/1244101224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103170 0x7fa9c41035f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009900 tx=0x7fa9b402bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/1244101224 shutdown_connections 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/1244101224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103b30 0x7fa9c410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/1244101224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103170 0x7fa9c41035f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/1244101224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 0x7fa9c41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/1244101224 >> 172.21.15.105:0/1244101224 conn(0x7fa9c40fd510 msgr2=0x7fa9c40ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/1244101224 shutdown_connections 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.551+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/1244101224 wait complete. 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 Processor -- start 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 -- start start 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 0x7fa9c41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103170 0x7fa9c41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410dac0 con 0x7fa9c4103b30 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410d940 con 0x7fa9c4103170 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9cb5f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410dc40 con 0x7fa9c4101f80 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49932/0 (socket says 172.21.15.105:49932) 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 -- 172.21.15.105:0/4159545347 learned_addr learned my addr 172.21.15.105:0/4159545347 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c8b8f640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103170 0x7fa9c41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9390640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 0x7fa9c41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 -- 172.21.15.105:0/4159545347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 msgr2=0x7fa9c41981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 0x7fa9c41981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 -- 172.21.15.105:0/4159545347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103170 msgr2=0x7fa9c41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103170 0x7fa9c41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 -- 172.21.15.105:0/4159545347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c419fdd0 con 0x7fa9c4103b30 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.552+0000 7fa9c9b91640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00efd0 tx=0x7fa9bc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.553+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc003b70 con 0x7fa9c4103b30 2024-04-22T06:35:46.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.553+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9bc003d10 con 0x7fa9c4103b30 2024-04-22T06:35:46.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.553+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc0108a0 con 0x7fa9c4103b30 2024-04-22T06:35:46.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.553+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c41a00c0 con 0x7fa9c4103b30 2024-04-22T06:35:46.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.553+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c41a0590 con 0x7fa9c4103b30 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.554+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c41093c0 con 0x7fa9c4103b30 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.555+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa9bc004b40 con 0x7fa9c4103b30 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.556+0000 7fa9ba7fc640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 0x7fa9b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.556+0000 7fa9c9390640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 0x7fa9b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.556+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fa9bc0d0ae0 con 0x7fa9c4103b30 2024-04-22T06:35:46.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.556+0000 7fa9c9390640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 0x7fa9b00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0005d00 tx=0x7fa9c0005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.560+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9bc09d610 con 0x7fa9c4103b30 2024-04-22T06:35:46.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.767+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa9c4062390 con 0x7fa9b00a3bb0 2024-04-22T06:35:46.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.772+0000 7fa9ba7fc640 1 -- 172.21.15.105:0/4159545347 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa9c4062390 con 0x7fa9b00a3bb0 2024-04-22T06:35:46.771 INFO:teuthology.orchestra.run.smithi105.stdout:/volumes/qa/sv_1/c5087255-4608-4544-90e9-8fe59b3f44f2 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 msgr2=0x7fa9b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 0x7fa9b00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0005d00 tx=0x7fa9c0005c50 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 msgr2=0x7fa9c419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00efd0 tx=0x7fa9bc00c6a0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 shutdown_connections 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9b00a3bb0 0x7fa9b00a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c4103b30 0x7fa9c419f760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9c4103170 0x7fa9c41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 --2- 172.21.15.105:0/4159545347 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9c4101f80 0x7fa9c41981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 >> 172.21.15.105:0/4159545347 conn(0x7fa9c40fd510 msgr2=0x7fa9c4106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 shutdown_connections 2024-04-22T06:35:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.774+0000 7fa9cb5f4640 1 -- 172.21.15.105:0/4159545347 wait complete. 2024-04-22T06:35:46.784 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:35:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 -- 172.21.15.105:0/4288475571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4100e70 msgr2=0x7fb1a4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/4288475571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4100e70 0x7fb1a4101290 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb194009860 tx=0x7fb19402bb70 comp rx=0 tx=0).stop 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 -- 172.21.15.105:0/4288475571 shutdown_connections 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/4288475571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4102980 0x7fb1a41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/4288475571 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4101fc0 0x7fb1a4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/4288475571 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4100e70 0x7fb1a4101290 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.882+0000 7fb1abd7e640 1 -- 172.21.15.105:0/4288475571 >> 172.21.15.105:0/4288475571 conn(0x7fb1a40fc3e0 msgr2=0x7fb1a40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- 172.21.15.105:0/4288475571 shutdown_connections 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- 172.21.15.105:0/4288475571 wait complete. 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 Processor -- start 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- start start 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4101fc0 0x7fb1a419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4102980 0x7fb1a419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410c970 con 0x7fb1a4100e70 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410c7f0 con 0x7fb1a4101fc0 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1abd7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a410caf0 con 0x7fb1a4102980 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1a9b1a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1a9b1a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49940/0 (socket says 172.21.15.105:49940) 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1a9b1a640 1 -- 172.21.15.105:0/2022659661 learned_addr learned my addr 172.21.15.105:0/2022659661 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.883+0000 7fb1a9319640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4101fc0 0x7fb1a419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1aa31b640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4102980 0x7fb1a419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 -- 172.21.15.105:0/2022659661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4102980 msgr2=0x7fb1a419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4102980 0x7fb1a419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 -- 172.21.15.105:0/2022659661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4101fc0 msgr2=0x7fb1a419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4101fc0 0x7fb1a419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 -- 172.21.15.105:0/2022659661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a419fe10 con 0x7fb1a4100e70 2024-04-22T06:35:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1a9b1a640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb194005940 tx=0x7fb1940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194041020 con 0x7fb1a4100e70 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a41a00a0 con 0x7fb1a4100e70 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1940035f0 con 0x7fb1a4100e70 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194039040 con 0x7fb1a4100e70 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.884+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41a0590 con 0x7fb1a4100e70 2024-04-22T06:35:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.885+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a41081f0 con 0x7fb1a4100e70 2024-04-22T06:35:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.887+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fb194044020 con 0x7fb1a4100e70 2024-04-22T06:35:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.887+0000 7fb19affd640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 0x7fb1900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.887+0000 7fb1a9319640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 0x7fb1900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.887+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fb1940f22b0 con 0x7fb1a4100e70 2024-04-22T06:35:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.887+0000 7fb1a9319640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 0x7fb1900a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40699e0 tx=0x7fb1a000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:46.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:46.891+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb19404b020 con 0x7fb1a4100e70 2024-04-22T06:35:47.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.044+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb194003aa0 con 0x7fb1a4100e70 2024-04-22T06:35:47.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.114+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1a402d5c0 con 0x7fb1a4100e70 2024-04-22T06:35:47.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.115+0000 7fb19affd640 1 -- 172.21.15.105:0/2022659661 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7fb1940bede0 con 0x7fb1a4100e70 2024-04-22T06:35:47.113 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:47.113 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:35:47.114 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:35:47.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 msgr2=0x7fb1900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 0x7fb1900a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40699e0 tx=0x7fb1a000b390 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 msgr2=0x7fb1a4069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb194005940 tx=0x7fb1940042f0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 shutdown_connections 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1900a3bb0 0x7fb1900a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1a4102980 0x7fb1a419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1a4101fc0 0x7fb1a419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 --2- 172.21.15.105:0/2022659661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a4100e70 0x7fb1a4069c70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 >> 172.21.15.105:0/2022659661 conn(0x7fb1a40fc3e0 msgr2=0x7fb1a41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 shutdown_connections 2024-04-22T06:35:47.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.117+0000 7fb1abd7e640 1 -- 172.21.15.105:0/2022659661 wait complete. 2024-04-22T06:35:47.125 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.125 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.125 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s wr, 13 op/s 2024-04-22T06:35:47.125 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:active 2024-04-22T06:35:47.126 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-22T06:35:47.126 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:47 smithi105 ceph-mon[27916]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:35:47.127 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 -- 172.21.15.105:0/1791560632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 msgr2=0x7fa5d8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 --2- 172.21.15.105:0/1791560632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8103610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009900 tx=0x7fa5cc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 -- 172.21.15.105:0/1791560632 shutdown_connections 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 --2- 172.21.15.105:0/1791560632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 0x7fa5d810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 --2- 172.21.15.105:0/1791560632 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8103610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 --2- 172.21.15.105:0/1791560632 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.227 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 -- 172.21.15.105:0/1791560632 >> 172.21.15.105:0/1791560632 conn(0x7fa5d80fd530 msgr2=0x7fa5d80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 -- 172.21.15.105:0/1791560632 shutdown_connections 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.226+0000 7fa5de30d640 1 -- 172.21.15.105:0/1791560632 wait complete. 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 Processor -- start 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 -- start start 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 0x7fa5d819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810db20 con 0x7fa5d8103190 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810d9a0 con 0x7fa5d8101fa0 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5de30d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d810dca0 con 0x7fa5d8103b50 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5d77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49954/0 (socket says 172.21.15.105:49954) 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5d77fe640 1 -- 172.21.15.105:0/515863208 learned_addr learned my addr 172.21.15.105:0/515863208 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:47.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.227+0000 7fa5d7fff640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5dc8aa640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 0x7fa5d819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 -- 172.21.15.105:0/515863208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 msgr2=0x7fa5d819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 0x7fa5d819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 -- 172.21.15.105:0/515863208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 msgr2=0x7fa5d81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 -- 172.21.15.105:0/515863208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d819fe10 con 0x7fa5d8103190 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d7fff640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d77fe640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc043020 con 0x7fa5d8103190 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5cc004440 con 0x7fa5d8103190 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d81a00a0 con 0x7fa5d8103190 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003bf0 con 0x7fa5d8103190 2024-04-22T06:35:47.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.228+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d81a0670 con 0x7fa5d8103190 2024-04-22T06:35:47.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.229+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d8109340 con 0x7fa5d8103190 2024-04-22T06:35:47.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.230+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa5cc046020 con 0x7fa5d8103190 2024-04-22T06:35:47.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.231+0000 7fa5d57fa640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 0x7fa5c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.231+0000 7fa5d7fff640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 0x7fa5c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.231+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fa5cc0f2690 con 0x7fa5d8103190 2024-04-22T06:35:47.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.231+0000 7fa5d7fff640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 0x7fa5c00a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8102ff0 tx=0x7fa5c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.235+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5cc04d020 con 0x7fa5d8103190 2024-04-22T06:35:47.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.413+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5d802d810 con 0x7fa5d8103190 2024-04-22T06:35:47.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.415+0000 7fa5d57fa640 1 -- 172.21.15.105:0/515863208 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22265 (secure 0 0 0) 0x7fa5cc0bf1d0 con 0x7fa5d8103190 2024-04-22T06:35:47.413 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:47.414 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":89,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:35:41.801359+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:47.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 msgr2=0x7fa5c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 0x7fa5c00a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8102ff0 tx=0x7fa5c8005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:47.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 msgr2=0x7fa5d8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc004040 comp rx=0 tx=0).stop 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 shutdown_connections 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5c00a3c40 0x7fa5c00a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5d8103b50 0x7fa5d819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5d8103190 0x7fa5d8198730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 --2- 172.21.15.105:0/515863208 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5d8101fa0 0x7fa5d81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 >> 172.21.15.105:0/515863208 conn(0x7fa5d80fd530 msgr2=0x7fa5d8106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 shutdown_connections 2024-04-22T06:35:47.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.417+0000 7fa5de30d640 1 -- 172.21.15.105:0/515863208 wait complete. 2024-04-22T06:35:47.427 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-22T06:35:47.427 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:35:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s wr, 13 op/s 2024-04-22T06:35:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:active 2024-04-22T06:35:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-22T06:35:47.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:47 smithi112 ceph-mon[31236]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s wr, 13 op/s 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] up:active 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-22T06:35:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:47 smithi177 ceph-mon[31346]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- 172.21.15.105:0/4028628172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 msgr2=0x7f979410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- 172.21.15.105:0/4028628172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979410a3e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f978c009ca0 tx=0x7f978c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- 172.21.15.105:0/4028628172 shutdown_connections 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- 172.21.15.105:0/4028628172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979410a3e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- 172.21.15.105:0/4028628172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 0x7f9794103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- 172.21.15.105:0/4028628172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f97941023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- 172.21.15.105:0/4028628172 >> 172.21.15.105:0/4028628172 conn(0x7f97940fd530 msgr2=0x7f97940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- 172.21.15.105:0/4028628172 shutdown_connections 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- 172.21.15.105:0/4028628172 wait complete. 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 Processor -- start 2024-04-22T06:35:47.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- start start 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f9794198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 0x7f97941986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f979410dae0 con 0x7f9794103b50 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f979410d960 con 0x7f9794101fa0 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.528+0000 7f979bc6a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f979410dc60 con 0x7f9794103190 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49976/0 (socket says 172.21.15.105:49976) 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 -- 172.21.15.105:0/1901868331 learned_addr learned my addr 172.21.15.105:0/1901868331 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f9799a06640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f9794198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f9799205640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 0x7f97941986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 -- 172.21.15.105:0/1901868331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 msgr2=0x7f97941986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 0x7f97941986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 -- 172.21.15.105:0/1901868331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 msgr2=0x7f9794198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f9794198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 -- 172.21.15.105:0/1901868331 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979419fd80 con 0x7f9794103b50 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f9799a06640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f9794198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979a207640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f978c005e30 tx=0x7f978c0346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c041020 con 0x7f9794103b50 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f978c034ca0 con 0x7f9794103b50 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97941a0010 con 0x7f9794103b50 2024-04-22T06:35:47.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c039040 con 0x7f9794103b50 2024-04-22T06:35:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.529+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97941a0500 con 0x7f9794103b50 2024-04-22T06:35:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.530+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97941092e0 con 0x7f9794103b50 2024-04-22T06:35:47.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.531+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f978c044020 con 0x7f9794103b50 2024-04-22T06:35:47.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.532+0000 7f978affd640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 0x7f97800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.532+0000 7f9799a06640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 0x7f97800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.532+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f978c0f1f70 con 0x7f9794103b50 2024-04-22T06:35:47.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.532+0000 7f9799a06640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 0x7f97800a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9794102ff0 tx=0x7f9784005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.536+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f978c04b020 con 0x7f9794103b50 2024-04-22T06:35:47.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.711+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9794062390 con 0x7f97800a3c40 2024-04-22T06:35:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.719+0000 7f978affd640 1 -- 172.21.15.105:0/1901868331 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258751 (secure 0 0 0) 0x7f9794062390 con 0x7f97800a3c40 2024-04-22T06:35:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:47.722 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 msgr2=0x7f97800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 0x7f97800a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9794102ff0 tx=0x7f9784005f90 comp rx=0 tx=0).stop 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 msgr2=0x7f979419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f978c005e30 tx=0x7f978c0346e0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 shutdown_connections 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f97800a3c40 0x7f97800a6100 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9794103b50 0x7f979419f710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9794103190 0x7f97941986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 --2- 172.21.15.105:0/1901868331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9794101fa0 0x7f9794198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 >> 172.21.15.105:0/1901868331 conn(0x7f97940fd530 msgr2=0x7f97941064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 shutdown_connections 2024-04-22T06:35:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.722+0000 7f979bc6a640 1 -- 172.21.15.105:0/1901868331 wait complete. 2024-04-22T06:35:47.731 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-22T06:35:46.620771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596010,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":275,"num_write_kb":2969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":205,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370660,"kb_used_data":13112,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571164,"statfs":{"total":1151940427776,"available":1151560871936,"internally_reserved":0,"allocated":13426688,"data_stored":7267964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5642,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.831320"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.809137+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:35:40.809137+0000","last_peered":"2024-04-22T06:35:40.809137+0000","last_clean":"2024-04-22T06:35:40.809137+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:35:40.809137+0000","last_undegraded":"2024-04-22T06:35:40.809137+0000","last_fullsized":"2024-04-22T06:35:40.809137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:42.766802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077823+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:35:42.077823+0000","last_peered":"2024-04-22T06:35:42.077823+0000","last_clean":"2024-04-22T06:35:42.077823+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:35:42.077823+0000","last_undegraded":"2024-04-22T06:35:42.077823+0000","last_fullsized":"2024-04-22T06:35:42.077823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077755+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:35:42.077755+0000","last_peered":"2024-04-22T06:35:42.077755+0000","last_clean":"2024-04-22T06:35:42.077755+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:35:42.077755+0000","last_undegraded":"2024-04-22T06:35:42.077755+0000","last_fullsized":"2024-04-22T06:35:42.077755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078492+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:35:42.078492+0000","last_peered":"2024-04-22T06:35:42.078492+0000","last_clean":"2024-04-22T06:35:42.078492+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:35:42.078492+0000","last_undegraded":"2024-04-22T06:35:42.078492+0000","last_fullsized":"2024-04-22T06:35:42.078492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813794+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:35:41.813794+0000","last_peered":"2024-04-22T06:35:41.813794+0000","last_clean":"2024-04-22T06:35:41.813794+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:35:41.813794+0000","last_undegraded":"2024-04-22T06:35:41.813794+0000","last_fullsized":"2024-04-22T06:35:41.813794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811448+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:35:41.811448+0000","last_peered":"2024-04-22T06:35:41.811448+0000","last_clean":"2024-04-22T06:35:41.811448+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:35:41.811448+0000","last_undegraded":"2024-04-22T06:35:41.811448+0000","last_fullsized":"2024-04-22T06:35:41.811448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947574+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:35:41.947574+0000","last_peered":"2024-04-22T06:35:41.947574+0000","last_clean":"2024-04-22T06:35:41.947574+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:35:41.947574+0000","last_undegraded":"2024-04-22T06:35:41.947574+0000","last_fullsized":"2024-04-22T06:35:41.947574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813128+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:35:41.813128+0000","last_peered":"2024-04-22T06:35:41.813128+0000","last_clean":"2024-04-22T06:35:41.813128+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:35:41.813128+0000","last_undegraded":"2024-04-22T06:35:41.813128+0000","last_fullsized":"2024-04-22T06:35:41.813128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.817089+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:35:40.817089+0000","last_peered":"2024-04-22T06:35:40.817089+0000","last_clean":"2024-04-22T06:35:40.817089+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:35:40.817089+0000","last_undegraded":"2024-04-22T06:35:40.817089+0000","last_fullsized":"2024-04-22T06:35:40.817089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:43.516389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811867+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:35:41.811867+0000","last_peered":"2024-04-22T06:35:41.811867+0000","last_clean":"2024-04-22T06:35:41.811867+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:35:41.811867+0000","last_undegraded":"2024-04-22T06:35:41.811867+0000","last_fullsized":"2024-04-22T06:35:41.811867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947867+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:35:41.947867+0000","last_peered":"2024-04-22T06:35:41.947867+0000","last_clean":"2024-04-22T06:35:41.947867+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:35:41.947867+0000","last_undegraded":"2024-04-22T06:35:41.947867+0000","last_fullsized":"2024-04-22T06:35:41.947867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078575+0000","last_change":"2024-04-22T06:35:40.812579+0000","last_active":"2024-04-22T06:35:42.078575+0000","last_peered":"2024-04-22T06:35:42.078575+0000","last_clean":"2024-04-22T06:35:42.078575+0000","last_became_active":"2024-04-22T06:35:40.812219+0000","last_became_peered":"2024-04-22T06:35:40.812219+0000","last_unstale":"2024-04-22T06:35:42.078575+0000","last_undegraded":"2024-04-22T06:35:42.078575+0000","last_fullsized":"2024-04-22T06:35:42.078575+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:43:04.137880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078072+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:35:42.078072+0000","last_peered":"2024-04-22T06:35:42.078072+0000","last_clean":"2024-04-22T06:35:42.078072+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:35:42.078072+0000","last_undegraded":"2024-04-22T06:35:42.078072+0000","last_fullsized":"2024-04-22T06:35:42.078072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077610+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:35:42.077610+0000","last_peered":"2024-04-22T06:35:42.077610+0000","last_clean":"2024-04-22T06:35:42.077610+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:35:42.077610+0000","last_undegraded":"2024-04-22T06:35:42.077610+0000","last_fullsized":"2024-04-22T06:35:42.077610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077369+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:35:42.077369+0000","last_peered":"2024-04-22T06:35:42.077369+0000","last_clean":"2024-04-22T06:35:42.077369+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:35:42.077369+0000","last_undegraded":"2024-04-22T06:35:42.077369+0000","last_fullsized":"2024-04-22T06:35:42.077369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819111+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:35:40.819111+0000","last_peered":"2024-04-22T06:35:40.819111+0000","last_clean":"2024-04-22T06:35:40.819111+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:35:40.819111+0000","last_undegraded":"2024-04-22T06:35:40.819111+0000","last_fullsized":"2024-04-22T06:35:40.819111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:42.902126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.807238+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:35:40.807238+0000","last_peered":"2024-04-22T06:35:40.807238+0000","last_clean":"2024-04-22T06:35:40.807238+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:35:40.807238+0000","last_undegraded":"2024-04-22T06:35:40.807238+0000","last_fullsized":"2024-04-22T06:35:40.807238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:45.692520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810619+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:35:41.810619+0000","last_peered":"2024-04-22T06:35:41.810619+0000","last_clean":"2024-04-22T06:35:41.810619+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:35:41.810619+0000","last_undegraded":"2024-04-22T06:35:41.810619+0000","last_fullsized":"2024-04-22T06:35:41.810619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812671+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:35:41.812671+0000","last_peered":"2024-04-22T06:35:41.812671+0000","last_clean":"2024-04-22T06:35:41.812671+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:35:41.812671+0000","last_undegraded":"2024-04-22T06:35:41.812671+0000","last_fullsized":"2024-04-22T06:35:41.812671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810643+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:35:41.810643+0000","last_peered":"2024-04-22T06:35:41.810643+0000","last_clean":"2024-04-22T06:35:41.810643+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:35:41.810643+0000","last_undegraded":"2024-04-22T06:35:41.810643+0000","last_fullsized":"2024-04-22T06:35:41.810643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813579+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:35:41.813579+0000","last_peered":"2024-04-22T06:35:41.813579+0000","last_clean":"2024-04-22T06:35:41.813579+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:35:41.813579+0000","last_undegraded":"2024-04-22T06:35:41.813579+0000","last_fullsized":"2024-04-22T06:35:41.813579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813586+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:35:41.813586+0000","last_peered":"2024-04-22T06:35:41.813586+0000","last_clean":"2024-04-22T06:35:41.813586+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:35:41.813586+0000","last_undegraded":"2024-04-22T06:35:41.813586+0000","last_fullsized":"2024-04-22T06:35:41.813586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077838+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:35:42.077838+0000","last_peered":"2024-04-22T06:35:42.077838+0000","last_clean":"2024-04-22T06:35:42.077838+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:35:42.077838+0000","last_undegraded":"2024-04-22T06:35:42.077838+0000","last_fullsized":"2024-04-22T06:35:42.077838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814193+0000","last_change":"2024-04-22T06:35:39.805656+0000","last_active":"2024-04-22T06:35:41.814193+0000","last_peered":"2024-04-22T06:35:41.814193+0000","last_clean":"2024-04-22T06:35:41.814193+0000","last_became_active":"2024-04-22T06:35:39.805329+0000","last_became_peered":"2024-04-22T06:35:39.805329+0000","last_unstale":"2024-04-22T06:35:41.814193+0000","last_undegraded":"2024-04-22T06:35:41.814193+0000","last_fullsized":"2024-04-22T06:35:41.814193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:18.405514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820965+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:35:40.820965+0000","last_peered":"2024-04-22T06:35:40.820965+0000","last_clean":"2024-04-22T06:35:40.820965+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:35:40.820965+0000","last_undegraded":"2024-04-22T06:35:40.820965+0000","last_fullsized":"2024-04-22T06:35:40.820965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:37.840486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814125+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:35:41.814125+0000","last_peered":"2024-04-22T06:35:41.814125+0000","last_clean":"2024-04-22T06:35:41.814125+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:35:41.814125+0000","last_undegraded":"2024-04-22T06:35:41.814125+0000","last_fullsized":"2024-04-22T06:35:41.814125+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078527+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:35:42.078527+0000","last_peered":"2024-04-22T06:35:42.078527+0000","last_clean":"2024-04-22T06:35:42.078527+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:35:42.078527+0000","last_undegraded":"2024-04-22T06:35:42.078527+0000","last_fullsized":"2024-04-22T06:35:42.078527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.029160+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:35:43.029160+0000","last_peered":"2024-04-22T06:35:43.029160+0000","last_clean":"2024-04-22T06:35:43.029160+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:35:43.029160+0000","last_undegraded":"2024-04-22T06:35:43.029160+0000","last_fullsized":"2024-04-22T06:35:43.029160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813862+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:35:41.813862+0000","last_peered":"2024-04-22T06:35:41.813862+0000","last_clean":"2024-04-22T06:35:41.813862+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:35:41.813862+0000","last_undegraded":"2024-04-22T06:35:41.813862+0000","last_fullsized":"2024-04-22T06:35:41.813862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811335+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:35:41.811335+0000","last_peered":"2024-04-22T06:35:41.811335+0000","last_clean":"2024-04-22T06:35:41.811335+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:35:41.811335+0000","last_undegraded":"2024-04-22T06:35:41.811335+0000","last_fullsized":"2024-04-22T06:35:41.811335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078581+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:35:42.078581+0000","last_peered":"2024-04-22T06:35:42.078581+0000","last_clean":"2024-04-22T06:35:42.078581+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:35:42.078581+0000","last_undegraded":"2024-04-22T06:35:42.078581+0000","last_fullsized":"2024-04-22T06:35:42.078581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.033692+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:35:44.033692+0000","last_peered":"2024-04-22T06:35:44.033692+0000","last_clean":"2024-04-22T06:35:44.033692+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:35:44.033692+0000","last_undegraded":"2024-04-22T06:35:44.033692+0000","last_fullsized":"2024-04-22T06:35:44.033692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813361+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:35:41.813361+0000","last_peered":"2024-04-22T06:35:41.813361+0000","last_clean":"2024-04-22T06:35:41.813361+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:35:41.813361+0000","last_undegraded":"2024-04-22T06:35:41.813361+0000","last_fullsized":"2024-04-22T06:35:41.813361+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027345+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:35:43.027345+0000","last_peered":"2024-04-22T06:35:43.027345+0000","last_clean":"2024-04-22T06:35:43.027345+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:35:43.027345+0000","last_undegraded":"2024-04-22T06:35:43.027345+0000","last_fullsized":"2024-04-22T06:35:43.027345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810501+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:35:41.810501+0000","last_peered":"2024-04-22T06:35:41.810501+0000","last_clean":"2024-04-22T06:35:41.810501+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:35:41.810501+0000","last_undegraded":"2024-04-22T06:35:41.810501+0000","last_fullsized":"2024-04-22T06:35:41.810501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811795+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:35:41.811795+0000","last_peered":"2024-04-22T06:35:41.811795+0000","last_clean":"2024-04-22T06:35:41.811795+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:35:41.811795+0000","last_undegraded":"2024-04-22T06:35:41.811795+0000","last_fullsized":"2024-04-22T06:35:41.811795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.825426+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:35:40.825426+0000","last_peered":"2024-04-22T06:35:40.825426+0000","last_clean":"2024-04-22T06:35:40.825426+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:35:40.825426+0000","last_undegraded":"2024-04-22T06:35:40.825426+0000","last_fullsized":"2024-04-22T06:35:40.825426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:59:18.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027538+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:35:43.027538+0000","last_peered":"2024-04-22T06:35:43.027538+0000","last_clean":"2024-04-22T06:35:43.027538+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:35:43.027538+0000","last_undegraded":"2024-04-22T06:35:43.027538+0000","last_fullsized":"2024-04-22T06:35:43.027538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077847+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:35:42.077847+0000","last_peered":"2024-04-22T06:35:42.077847+0000","last_clean":"2024-04-22T06:35:42.077847+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:35:42.077847+0000","last_undegraded":"2024-04-22T06:35:42.077847+0000","last_fullsized":"2024-04-22T06:35:42.077847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.025736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:35:43.025736+0000","last_peered":"2024-04-22T06:35:43.025736+0000","last_clean":"2024-04-22T06:35:43.025736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:35:43.025736+0000","last_undegraded":"2024-04-22T06:35:43.025736+0000","last_fullsized":"2024-04-22T06:35:43.025736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820945+0000","last_change":"2024-04-22T06:35:40.817278+0000","last_active":"2024-04-22T06:35:40.820945+0000","last_peered":"2024-04-22T06:35:40.820945+0000","last_clean":"2024-04-22T06:35:40.820945+0000","last_became_active":"2024-04-22T06:35:40.816931+0000","last_became_peered":"2024-04-22T06:35:40.816931+0000","last_unstale":"2024-04-22T06:35:40.820945+0000","last_undegraded":"2024-04-22T06:35:40.820945+0000","last_fullsized":"2024-04-22T06:35:40.820945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:20.846914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811518+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:35:41.811518+0000","last_peered":"2024-04-22T06:35:41.811518+0000","last_clean":"2024-04-22T06:35:41.811518+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:35:41.811518+0000","last_undegraded":"2024-04-22T06:35:41.811518+0000","last_fullsized":"2024-04-22T06:35:41.811518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814031+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:35:41.814031+0000","last_peered":"2024-04-22T06:35:41.814031+0000","last_clean":"2024-04-22T06:35:41.814031+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:35:41.814031+0000","last_undegraded":"2024-04-22T06:35:41.814031+0000","last_fullsized":"2024-04-22T06:35:41.814031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.033836+0000","last_change":"2024-04-22T06:35:39.805809+0000","last_active":"2024-04-22T06:35:44.033836+0000","last_peered":"2024-04-22T06:35:44.033836+0000","last_clean":"2024-04-22T06:35:44.033836+0000","last_became_active":"2024-04-22T06:35:39.805308+0000","last_became_peered":"2024-04-22T06:35:39.805308+0000","last_unstale":"2024-04-22T06:35:44.033836+0000","last_undegraded":"2024-04-22T06:35:44.033836+0000","last_fullsized":"2024-04-22T06:35:44.033836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:00.335655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077982+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:35:42.077982+0000","last_peered":"2024-04-22T06:35:42.077982+0000","last_clean":"2024-04-22T06:35:42.077982+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:35:42.077982+0000","last_undegraded":"2024-04-22T06:35:42.077982+0000","last_fullsized":"2024-04-22T06:35:42.077982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027515+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:35:43.027515+0000","last_peered":"2024-04-22T06:35:43.027515+0000","last_clean":"2024-04-22T06:35:43.027515+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:35:43.027515+0000","last_undegraded":"2024-04-22T06:35:43.027515+0000","last_fullsized":"2024-04-22T06:35:43.027515+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947896+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:35:41.947896+0000","last_peered":"2024-04-22T06:35:41.947896+0000","last_clean":"2024-04-22T06:35:41.947896+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:35:41.947896+0000","last_undegraded":"2024-04-22T06:35:41.947896+0000","last_fullsized":"2024-04-22T06:35:41.947896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.043267+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:35:45.043267+0000","last_peered":"2024-04-22T06:35:45.043267+0000","last_clean":"2024-04-22T06:35:45.043267+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:35:45.043267+0000","last_undegraded":"2024-04-22T06:35:45.043267+0000","last_fullsized":"2024-04-22T06:35:45.043267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947416+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:35:41.947416+0000","last_peered":"2024-04-22T06:35:41.947416+0000","last_clean":"2024-04-22T06:35:41.947416+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:35:41.947416+0000","last_undegraded":"2024-04-22T06:35:41.947416+0000","last_fullsized":"2024-04-22T06:35:41.947416+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813620+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:35:41.813620+0000","last_peered":"2024-04-22T06:35:41.813620+0000","last_clean":"2024-04-22T06:35:41.813620+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:35:41.813620+0000","last_undegraded":"2024-04-22T06:35:41.813620+0000","last_fullsized":"2024-04-22T06:35:41.813620+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078182+0000","last_change":"2024-04-22T06:35:40.819992+0000","last_active":"2024-04-22T06:35:42.078182+0000","last_peered":"2024-04-22T06:35:42.078182+0000","last_clean":"2024-04-22T06:35:42.078182+0000","last_became_active":"2024-04-22T06:35:40.819652+0000","last_became_peered":"2024-04-22T06:35:40.819652+0000","last_unstale":"2024-04-22T06:35:42.078182+0000","last_undegraded":"2024-04-22T06:35:42.078182+0000","last_fullsized":"2024-04-22T06:35:42.078182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:34.515492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814151+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:35:41.814151+0000","last_peered":"2024-04-22T06:35:41.814151+0000","last_clean":"2024-04-22T06:35:41.814151+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:35:41.814151+0000","last_undegraded":"2024-04-22T06:35:41.814151+0000","last_fullsized":"2024-04-22T06:35:41.814151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819751+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:35:40.819751+0000","last_peered":"2024-04-22T06:35:40.819751+0000","last_clean":"2024-04-22T06:35:40.819751+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:35:40.819751+0000","last_undegraded":"2024-04-22T06:35:40.819751+0000","last_fullsized":"2024-04-22T06:35:40.819751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:02.228001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810642+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:35:41.810642+0000","last_peered":"2024-04-22T06:35:41.810642+0000","last_clean":"2024-04-22T06:35:41.810642+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:35:41.810642+0000","last_undegraded":"2024-04-22T06:35:41.810642+0000","last_fullsized":"2024-04-22T06:35:41.810642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811834+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:35:41.811834+0000","last_peered":"2024-04-22T06:35:41.811834+0000","last_clean":"2024-04-22T06:35:41.811834+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:35:41.811834+0000","last_undegraded":"2024-04-22T06:35:41.811834+0000","last_fullsized":"2024-04-22T06:35:41.811834+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947529+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:35:41.947529+0000","last_peered":"2024-04-22T06:35:41.947529+0000","last_clean":"2024-04-22T06:35:41.947529+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:35:41.947529+0000","last_undegraded":"2024-04-22T06:35:41.947529+0000","last_fullsized":"2024-04-22T06:35:41.947529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811623+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:35:41.811623+0000","last_peered":"2024-04-22T06:35:41.811623+0000","last_clean":"2024-04-22T06:35:41.811623+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:35:41.811623+0000","last_undegraded":"2024-04-22T06:35:41.811623+0000","last_fullsized":"2024-04-22T06:35:41.811623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078205+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:35:42.078205+0000","last_peered":"2024-04-22T06:35:42.078205+0000","last_clean":"2024-04-22T06:35:42.078205+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:35:42.078205+0000","last_undegraded":"2024-04-22T06:35:42.078205+0000","last_fullsized":"2024-04-22T06:35:42.078205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.035142+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:35:44.035142+0000","last_peered":"2024-04-22T06:35:44.035142+0000","last_clean":"2024-04-22T06:35:44.035142+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:35:44.035142+0000","last_undegraded":"2024-04-22T06:35:44.035142+0000","last_fullsized":"2024-04-22T06:35:44.035142+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813419+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:35:41.813419+0000","last_peered":"2024-04-22T06:35:41.813419+0000","last_clean":"2024-04-22T06:35:41.813419+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:35:41.813419+0000","last_undegraded":"2024-04-22T06:35:41.813419+0000","last_fullsized":"2024-04-22T06:35:41.813419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.044907+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:35:45.044907+0000","last_peered":"2024-04-22T06:35:45.044907+0000","last_clean":"2024-04-22T06:35:45.044907+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:35:45.044907+0000","last_undegraded":"2024-04-22T06:35:45.044907+0000","last_fullsized":"2024-04-22T06:35:45.044907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947311+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:35:41.947311+0000","last_peered":"2024-04-22T06:35:41.947311+0000","last_clean":"2024-04-22T06:35:41.947311+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:35:41.947311+0000","last_undegraded":"2024-04-22T06:35:41.947311+0000","last_fullsized":"2024-04-22T06:35:41.947311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077266+0000","last_change":"2024-04-22T06:35:40.816889+0000","last_active":"2024-04-22T06:35:42.077266+0000","last_peered":"2024-04-22T06:35:42.077266+0000","last_clean":"2024-04-22T06:35:42.077266+0000","last_became_active":"2024-04-22T06:35:40.816555+0000","last_became_peered":"2024-04-22T06:35:40.816555+0000","last_unstale":"2024-04-22T06:35:42.077266+0000","last_undegraded":"2024-04-22T06:35:42.077266+0000","last_fullsized":"2024-04-22T06:35:42.077266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:23.395675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027170+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:35:43.027170+0000","last_peered":"2024-04-22T06:35:43.027170+0000","last_clean":"2024-04-22T06:35:43.027170+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:35:43.027170+0000","last_undegraded":"2024-04-22T06:35:43.027170+0000","last_fullsized":"2024-04-22T06:35:43.027170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814047+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:35:41.814047+0000","last_peered":"2024-04-22T06:35:41.814047+0000","last_clean":"2024-04-22T06:35:41.814047+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:35:41.814047+0000","last_undegraded":"2024-04-22T06:35:41.814047+0000","last_fullsized":"2024-04-22T06:35:41.814047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078163+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:35:42.078163+0000","last_peered":"2024-04-22T06:35:42.078163+0000","last_clean":"2024-04-22T06:35:42.078163+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:35:42.078163+0000","last_undegraded":"2024-04-22T06:35:42.078163+0000","last_fullsized":"2024-04-22T06:35:42.078163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813414+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:35:41.813414+0000","last_peered":"2024-04-22T06:35:41.813414+0000","last_clean":"2024-04-22T06:35:41.813414+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:35:41.813414+0000","last_undegraded":"2024-04-22T06:35:41.813414+0000","last_fullsized":"2024-04-22T06:35:41.813414+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813655+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:35:41.813655+0000","last_peered":"2024-04-22T06:35:41.813655+0000","last_clean":"2024-04-22T06:35:41.813655+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:35:41.813655+0000","last_undegraded":"2024-04-22T06:35:41.813655+0000","last_fullsized":"2024-04-22T06:35:41.813655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.026959+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:35:43.026959+0000","last_peered":"2024-04-22T06:35:43.026959+0000","last_clean":"2024-04-22T06:35:43.026959+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:35:43.026959+0000","last_undegraded":"2024-04-22T06:35:43.026959+0000","last_fullsized":"2024-04-22T06:35:43.026959+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077854+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:35:42.077854+0000","last_peered":"2024-04-22T06:35:42.077854+0000","last_clean":"2024-04-22T06:35:42.077854+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:35:42.077854+0000","last_undegraded":"2024-04-22T06:35:42.077854+0000","last_fullsized":"2024-04-22T06:35:42.077854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.800396+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:35:40.800396+0000","last_peered":"2024-04-22T06:35:40.800396+0000","last_clean":"2024-04-22T06:35:40.800396+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:35:40.800396+0000","last_undegraded":"2024-04-22T06:35:40.800396+0000","last_fullsized":"2024-04-22T06:35:40.800396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813748+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:35:41.813748+0000","last_peered":"2024-04-22T06:35:41.813748+0000","last_clean":"2024-04-22T06:35:41.813748+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:35:41.813748+0000","last_undegraded":"2024-04-22T06:35:41.813748+0000","last_fullsized":"2024-04-22T06:35:41.813748+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.026069+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:35:43.026069+0000","last_peered":"2024-04-22T06:35:43.026069+0000","last_clean":"2024-04-22T06:35:43.026069+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:35:43.026069+0000","last_undegraded":"2024-04-22T06:35:43.026069+0000","last_fullsized":"2024-04-22T06:35:43.026069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.806899+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:35:40.806899+0000","last_peered":"2024-04-22T06:35:40.806899+0000","last_clean":"2024-04-22T06:35:40.806899+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:35:40.806899+0000","last_undegraded":"2024-04-22T06:35:40.806899+0000","last_fullsized":"2024-04-22T06:35:40.806899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:46.327532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.021959+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:35:44.021959+0000","last_peered":"2024-04-22T06:35:44.021959+0000","last_clean":"2024-04-22T06:35:44.021959+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:35:44.021959+0000","last_undegraded":"2024-04-22T06:35:44.021959+0000","last_fullsized":"2024-04-22T06:35:44.021959+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813291+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:35:41.813291+0000","last_peered":"2024-04-22T06:35:41.813291+0000","last_clean":"2024-04-22T06:35:41.813291+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:35:41.813291+0000","last_undegraded":"2024-04-22T06:35:41.813291+0000","last_fullsized":"2024-04-22T06:35:41.813291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813706+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:35:41.813706+0000","last_peered":"2024-04-22T06:35:41.813706+0000","last_clean":"2024-04-22T06:35:41.813706+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:35:41.813706+0000","last_undegraded":"2024-04-22T06:35:41.813706+0000","last_fullsized":"2024-04-22T06:35:41.813706+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947721+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:35:41.947721+0000","last_peered":"2024-04-22T06:35:41.947721+0000","last_clean":"2024-04-22T06:35:41.947721+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:35:41.947721+0000","last_undegraded":"2024-04-22T06:35:41.947721+0000","last_fullsized":"2024-04-22T06:35:41.947721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.045173+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:35:45.045173+0000","last_peered":"2024-04-22T06:35:45.045173+0000","last_clean":"2024-04-22T06:35:45.045173+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:35:45.045173+0000","last_undegraded":"2024-04-22T06:35:45.045173+0000","last_fullsized":"2024-04-22T06:35:45.045173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947921+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:35:41.947921+0000","last_peered":"2024-04-22T06:35:41.947921+0000","last_clean":"2024-04-22T06:35:41.947921+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:35:41.947921+0000","last_undegraded":"2024-04-22T06:35:41.947921+0000","last_fullsized":"2024-04-22T06:35:41.947921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078260+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:35:42.078260+0000","last_peered":"2024-04-22T06:35:42.078260+0000","last_clean":"2024-04-22T06:35:42.078260+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:35:42.078260+0000","last_undegraded":"2024-04-22T06:35:42.078260+0000","last_fullsized":"2024-04-22T06:35:42.078260+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947766+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:35:41.947766+0000","last_peered":"2024-04-22T06:35:41.947766+0000","last_clean":"2024-04-22T06:35:41.947766+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:35:41.947766+0000","last_undegraded":"2024-04-22T06:35:41.947766+0000","last_fullsized":"2024-04-22T06:35:41.947766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078116+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:35:42.078116+0000","last_peered":"2024-04-22T06:35:42.078116+0000","last_clean":"2024-04-22T06:35:42.078116+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:35:42.078116+0000","last_undegraded":"2024-04-22T06:35:42.078116+0000","last_fullsized":"2024-04-22T06:35:42.078116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947838+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:35:41.947838+0000","last_peered":"2024-04-22T06:35:41.947838+0000","last_clean":"2024-04-22T06:35:41.947838+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:35:41.947838+0000","last_undegraded":"2024-04-22T06:35:41.947838+0000","last_fullsized":"2024-04-22T06:35:41.947838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819755+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:35:40.819755+0000","last_peered":"2024-04-22T06:35:40.819755+0000","last_clean":"2024-04-22T06:35:40.819755+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:35:40.819755+0000","last_undegraded":"2024-04-22T06:35:40.819755+0000","last_fullsized":"2024-04-22T06:35:40.819755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:18:52.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812553+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:35:41.812553+0000","last_peered":"2024-04-22T06:35:41.812553+0000","last_clean":"2024-04-22T06:35:41.812553+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:35:41.812553+0000","last_undegraded":"2024-04-22T06:35:41.812553+0000","last_fullsized":"2024-04-22T06:35:41.812553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812235+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:35:41.812235+0000","last_peered":"2024-04-22T06:35:41.812235+0000","last_clean":"2024-04-22T06:35:41.812235+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:35:41.812235+0000","last_undegraded":"2024-04-22T06:35:41.812235+0000","last_fullsized":"2024-04-22T06:35:41.812235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077557+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:35:42.077557+0000","last_peered":"2024-04-22T06:35:42.077557+0000","last_clean":"2024-04-22T06:35:42.077557+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:35:42.077557+0000","last_undegraded":"2024-04-22T06:35:42.077557+0000","last_fullsized":"2024-04-22T06:35:42.077557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078299+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:35:42.078299+0000","last_peered":"2024-04-22T06:35:42.078299+0000","last_clean":"2024-04-22T06:35:42.078299+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:35:42.078299+0000","last_undegraded":"2024-04-22T06:35:42.078299+0000","last_fullsized":"2024-04-22T06:35:42.078299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.812166+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:35:40.812166+0000","last_peered":"2024-04-22T06:35:40.812166+0000","last_clean":"2024-04-22T06:35:40.812166+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:35:40.812166+0000","last_undegraded":"2024-04-22T06:35:40.812166+0000","last_fullsized":"2024-04-22T06:35:40.812166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:26.679625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812853+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:35:41.812853+0000","last_peered":"2024-04-22T06:35:41.812853+0000","last_clean":"2024-04-22T06:35:41.812853+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:35:41.812853+0000","last_undegraded":"2024-04-22T06:35:41.812853+0000","last_fullsized":"2024-04-22T06:35:41.812853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813655+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:35:41.813655+0000","last_peered":"2024-04-22T06:35:41.813655+0000","last_clean":"2024-04-22T06:35:41.813655+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:35:41.813655+0000","last_undegraded":"2024-04-22T06:35:41.813655+0000","last_fullsized":"2024-04-22T06:35:41.813655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077996+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:35:42.077996+0000","last_peered":"2024-04-22T06:35:42.077996+0000","last_clean":"2024-04-22T06:35:42.077996+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:35:42.077996+0000","last_undegraded":"2024-04-22T06:35:42.077996+0000","last_fullsized":"2024-04-22T06:35:42.077996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813198+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:35:41.813198+0000","last_peered":"2024-04-22T06:35:41.813198+0000","last_clean":"2024-04-22T06:35:41.813198+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:35:41.813198+0000","last_undegraded":"2024-04-22T06:35:41.813198+0000","last_fullsized":"2024-04-22T06:35:41.813198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820902+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:35:40.820902+0000","last_peered":"2024-04-22T06:35:40.820902+0000","last_clean":"2024-04-22T06:35:40.820902+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:35:40.820902+0000","last_undegraded":"2024-04-22T06:35:40.820902+0000","last_fullsized":"2024-04-22T06:35:40.820902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:44.395476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078171+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:35:42.078171+0000","last_peered":"2024-04-22T06:35:42.078171+0000","last_clean":"2024-04-22T06:35:42.078171+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:35:42.078171+0000","last_undegraded":"2024-04-22T06:35:42.078171+0000","last_fullsized":"2024-04-22T06:35:42.078171+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.809169+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:35:40.809169+0000","last_peered":"2024-04-22T06:35:40.809169+0000","last_clean":"2024-04-22T06:35:40.809169+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:35:40.809169+0000","last_undegraded":"2024-04-22T06:35:40.809169+0000","last_fullsized":"2024-04-22T06:35:40.809169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:31:02.737060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5642,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":12804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":449941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514514,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1576960,"data_stored":1050685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710744,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":962560,"data_stored":459093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972380,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":966656,"data_stored":459115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266721,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1564672,"data_stored":1050083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430247,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":458601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":958464,"data_stored":451553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":459205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":461473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888112,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":459155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691882,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1581056,"data_stored":1050355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:35:47.736 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:35:47.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- 172.21.15.105:0/649601748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 msgr2=0x7fecd81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- 172.21.15.105:0/649601748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81092a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009900 tx=0x7fecc002bc30 comp rx=0 tx=0).stop 2024-04-22T06:35:47.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- 172.21.15.105:0/649601748 shutdown_connections 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- 172.21.15.105:0/649601748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81092a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- 172.21.15.105:0/649601748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8101fc0 0x7fecd8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- 172.21.15.105:0/649601748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecd8100e70 0x7fecd8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- 172.21.15.105:0/649601748 >> 172.21.15.105:0/649601748 conn(0x7fecd80fc3e0 msgr2=0x7fecd80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- 172.21.15.105:0/649601748 shutdown_connections 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- 172.21.15.105:0/649601748 wait complete. 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 Processor -- start 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- start start 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8100e70 0x7fecd819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecd8101fc0 0x7fecd819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fecd810ca20 con 0x7fecd8102980 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fecd810c8a0 con 0x7fecd8100e70 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.872+0000 7fecdec3f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fecd810cba0 con 0x7fecd8101fc0 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49998/0 (socket says 172.21.15.105:49998) 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 -- 172.21.15.105:0/1384378851 learned_addr learned my addr 172.21.15.105:0/1384378851 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdc9db640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8100e70 0x7fecd819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 -- 172.21.15.105:0/1384378851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecd8101fc0 msgr2=0x7fecd819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecd8101fc0 0x7fecd819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 -- 172.21.15.105:0/1384378851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8100e70 msgr2=0x7fecd819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8100e70 0x7fecd819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 -- 172.21.15.105:0/1384378851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd81a1fe0 con 0x7fecd8102980 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdd1dc640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009230 tx=0x7fecc00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0043020 con 0x7fecd8102980 2024-04-22T06:35:47.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecc00044c0 con 0x7fecd8102980 2024-04-22T06:35:47.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd81a2270 con 0x7fecd8102980 2024-04-22T06:35:47.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0003bc0 con 0x7fecd8102980 2024-04-22T06:35:47.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.873+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd81a2890 con 0x7fecd8102980 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.874+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd81a2a90 con 0x7fecd8102980 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.875+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fecc0046020 con 0x7fecd8102980 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.876+0000 7fecd5ffb640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 0x7fecc40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.876+0000 7fecdc9db640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 0x7fecc40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.876+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fecc00f25a0 con 0x7fecd8102980 2024-04-22T06:35:47.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.876+0000 7fecdc9db640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 0x7fecc40a6180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fecd000a400 tx=0x7fecd000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:47.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:47.880+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fecc004d020 con 0x7fecd8102980 2024-04-22T06:35:48.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.056+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fecd81a2c60 con 0x7fecc40a3cc0 2024-04-22T06:35:48.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.063+0000 7fecd5ffb640 1 -- 172.21.15.105:0/1384378851 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258751 (secure 0 0 0) 0x7fecd81a2c60 con 0x7fecc40a3cc0 2024-04-22T06:35:48.067 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:48.068 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 msgr2=0x7fecc40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 0x7fecc40a6180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fecd000a400 tx=0x7fecd000d040 comp rx=0 tx=0).stop 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 msgr2=0x7fecd81a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009230 tx=0x7fecc00040c0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 shutdown_connections 2024-04-22T06:35:48.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fecc40a3cc0 0x7fecc40a6180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecd8102980 0x7fecd81a1970 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fecd8101fc0 0x7fecd819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 --2- 172.21.15.105:0/1384378851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecd8100e70 0x7fecd819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 >> 172.21.15.105:0/1384378851 conn(0x7fecd80fc3e0 msgr2=0x7fecd80fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 shutdown_connections 2024-04-22T06:35:48.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:48.066+0000 7fecdec3f640 1 -- 172.21.15.105:0/1384378851 wait complete. 2024-04-22T06:35:48.076 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-22T06:35:46.620771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596010,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":275,"num_write_kb":2969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":205,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370660,"kb_used_data":13112,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571164,"statfs":{"total":1151940427776,"available":1151560871936,"internally_reserved":0,"allocated":13426688,"data_stored":7267964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5642,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.831320"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.809137+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:35:40.809137+0000","last_peered":"2024-04-22T06:35:40.809137+0000","last_clean":"2024-04-22T06:35:40.809137+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:35:40.809137+0000","last_undegraded":"2024-04-22T06:35:40.809137+0000","last_fullsized":"2024-04-22T06:35:40.809137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:42.766802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077823+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:35:42.077823+0000","last_peered":"2024-04-22T06:35:42.077823+0000","last_clean":"2024-04-22T06:35:42.077823+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:35:42.077823+0000","last_undegraded":"2024-04-22T06:35:42.077823+0000","last_fullsized":"2024-04-22T06:35:42.077823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077755+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:35:42.077755+0000","last_peered":"2024-04-22T06:35:42.077755+0000","last_clean":"2024-04-22T06:35:42.077755+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:35:42.077755+0000","last_undegraded":"2024-04-22T06:35:42.077755+0000","last_fullsized":"2024-04-22T06:35:42.077755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078492+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:35:42.078492+0000","last_peered":"2024-04-22T06:35:42.078492+0000","last_clean":"2024-04-22T06:35:42.078492+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:35:42.078492+0000","last_undegraded":"2024-04-22T06:35:42.078492+0000","last_fullsized":"2024-04-22T06:35:42.078492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813794+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:35:41.813794+0000","last_peered":"2024-04-22T06:35:41.813794+0000","last_clean":"2024-04-22T06:35:41.813794+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:35:41.813794+0000","last_undegraded":"2024-04-22T06:35:41.813794+0000","last_fullsized":"2024-04-22T06:35:41.813794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811448+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:35:41.811448+0000","last_peered":"2024-04-22T06:35:41.811448+0000","last_clean":"2024-04-22T06:35:41.811448+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:35:41.811448+0000","last_undegraded":"2024-04-22T06:35:41.811448+0000","last_fullsized":"2024-04-22T06:35:41.811448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947574+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:35:41.947574+0000","last_peered":"2024-04-22T06:35:41.947574+0000","last_clean":"2024-04-22T06:35:41.947574+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:35:41.947574+0000","last_undegraded":"2024-04-22T06:35:41.947574+0000","last_fullsized":"2024-04-22T06:35:41.947574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813128+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:35:41.813128+0000","last_peered":"2024-04-22T06:35:41.813128+0000","last_clean":"2024-04-22T06:35:41.813128+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:35:41.813128+0000","last_undegraded":"2024-04-22T06:35:41.813128+0000","last_fullsized":"2024-04-22T06:35:41.813128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.817089+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:35:40.817089+0000","last_peered":"2024-04-22T06:35:40.817089+0000","last_clean":"2024-04-22T06:35:40.817089+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:35:40.817089+0000","last_undegraded":"2024-04-22T06:35:40.817089+0000","last_fullsized":"2024-04-22T06:35:40.817089+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:20:43.516389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811867+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:35:41.811867+0000","last_peered":"2024-04-22T06:35:41.811867+0000","last_clean":"2024-04-22T06:35:41.811867+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:35:41.811867+0000","last_undegraded":"2024-04-22T06:35:41.811867+0000","last_fullsized":"2024-04-22T06:35:41.811867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947867+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:35:41.947867+0000","last_peered":"2024-04-22T06:35:41.947867+0000","last_clean":"2024-04-22T06:35:41.947867+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:35:41.947867+0000","last_undegraded":"2024-04-22T06:35:41.947867+0000","last_fullsized":"2024-04-22T06:35:41.947867+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078575+0000","last_change":"2024-04-22T06:35:40.812579+0000","last_active":"2024-04-22T06:35:42.078575+0000","last_peered":"2024-04-22T06:35:42.078575+0000","last_clean":"2024-04-22T06:35:42.078575+0000","last_became_active":"2024-04-22T06:35:40.812219+0000","last_became_peered":"2024-04-22T06:35:40.812219+0000","last_unstale":"2024-04-22T06:35:42.078575+0000","last_undegraded":"2024-04-22T06:35:42.078575+0000","last_fullsized":"2024-04-22T06:35:42.078575+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:43:04.137880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078072+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:35:42.078072+0000","last_peered":"2024-04-22T06:35:42.078072+0000","last_clean":"2024-04-22T06:35:42.078072+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:35:42.078072+0000","last_undegraded":"2024-04-22T06:35:42.078072+0000","last_fullsized":"2024-04-22T06:35:42.078072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077610+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:35:42.077610+0000","last_peered":"2024-04-22T06:35:42.077610+0000","last_clean":"2024-04-22T06:35:42.077610+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:35:42.077610+0000","last_undegraded":"2024-04-22T06:35:42.077610+0000","last_fullsized":"2024-04-22T06:35:42.077610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077369+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:35:42.077369+0000","last_peered":"2024-04-22T06:35:42.077369+0000","last_clean":"2024-04-22T06:35:42.077369+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:35:42.077369+0000","last_undegraded":"2024-04-22T06:35:42.077369+0000","last_fullsized":"2024-04-22T06:35:42.077369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819111+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:35:40.819111+0000","last_peered":"2024-04-22T06:35:40.819111+0000","last_clean":"2024-04-22T06:35:40.819111+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:35:40.819111+0000","last_undegraded":"2024-04-22T06:35:40.819111+0000","last_fullsized":"2024-04-22T06:35:40.819111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:56:42.902126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.807238+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:35:40.807238+0000","last_peered":"2024-04-22T06:35:40.807238+0000","last_clean":"2024-04-22T06:35:40.807238+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:35:40.807238+0000","last_undegraded":"2024-04-22T06:35:40.807238+0000","last_fullsized":"2024-04-22T06:35:40.807238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:45.692520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810619+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:35:41.810619+0000","last_peered":"2024-04-22T06:35:41.810619+0000","last_clean":"2024-04-22T06:35:41.810619+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:35:41.810619+0000","last_undegraded":"2024-04-22T06:35:41.810619+0000","last_fullsized":"2024-04-22T06:35:41.810619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812671+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:35:41.812671+0000","last_peered":"2024-04-22T06:35:41.812671+0000","last_clean":"2024-04-22T06:35:41.812671+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:35:41.812671+0000","last_undegraded":"2024-04-22T06:35:41.812671+0000","last_fullsized":"2024-04-22T06:35:41.812671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810643+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:35:41.810643+0000","last_peered":"2024-04-22T06:35:41.810643+0000","last_clean":"2024-04-22T06:35:41.810643+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:35:41.810643+0000","last_undegraded":"2024-04-22T06:35:41.810643+0000","last_fullsized":"2024-04-22T06:35:41.810643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813579+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:35:41.813579+0000","last_peered":"2024-04-22T06:35:41.813579+0000","last_clean":"2024-04-22T06:35:41.813579+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:35:41.813579+0000","last_undegraded":"2024-04-22T06:35:41.813579+0000","last_fullsized":"2024-04-22T06:35:41.813579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813586+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:35:41.813586+0000","last_peered":"2024-04-22T06:35:41.813586+0000","last_clean":"2024-04-22T06:35:41.813586+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:35:41.813586+0000","last_undegraded":"2024-04-22T06:35:41.813586+0000","last_fullsized":"2024-04-22T06:35:41.813586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077838+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:35:42.077838+0000","last_peered":"2024-04-22T06:35:42.077838+0000","last_clean":"2024-04-22T06:35:42.077838+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:35:42.077838+0000","last_undegraded":"2024-04-22T06:35:42.077838+0000","last_fullsized":"2024-04-22T06:35:42.077838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814193+0000","last_change":"2024-04-22T06:35:39.805656+0000","last_active":"2024-04-22T06:35:41.814193+0000","last_peered":"2024-04-22T06:35:41.814193+0000","last_clean":"2024-04-22T06:35:41.814193+0000","last_became_active":"2024-04-22T06:35:39.805329+0000","last_became_peered":"2024-04-22T06:35:39.805329+0000","last_unstale":"2024-04-22T06:35:41.814193+0000","last_undegraded":"2024-04-22T06:35:41.814193+0000","last_fullsized":"2024-04-22T06:35:41.814193+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:16:18.405514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820965+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:35:40.820965+0000","last_peered":"2024-04-22T06:35:40.820965+0000","last_clean":"2024-04-22T06:35:40.820965+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:35:40.820965+0000","last_undegraded":"2024-04-22T06:35:40.820965+0000","last_fullsized":"2024-04-22T06:35:40.820965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:45:37.840486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814125+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:35:41.814125+0000","last_peered":"2024-04-22T06:35:41.814125+0000","last_clean":"2024-04-22T06:35:41.814125+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:35:41.814125+0000","last_undegraded":"2024-04-22T06:35:41.814125+0000","last_fullsized":"2024-04-22T06:35:41.814125+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078527+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:35:42.078527+0000","last_peered":"2024-04-22T06:35:42.078527+0000","last_clean":"2024-04-22T06:35:42.078527+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:35:42.078527+0000","last_undegraded":"2024-04-22T06:35:42.078527+0000","last_fullsized":"2024-04-22T06:35:42.078527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.029160+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:35:43.029160+0000","last_peered":"2024-04-22T06:35:43.029160+0000","last_clean":"2024-04-22T06:35:43.029160+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:35:43.029160+0000","last_undegraded":"2024-04-22T06:35:43.029160+0000","last_fullsized":"2024-04-22T06:35:43.029160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813862+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:35:41.813862+0000","last_peered":"2024-04-22T06:35:41.813862+0000","last_clean":"2024-04-22T06:35:41.813862+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:35:41.813862+0000","last_undegraded":"2024-04-22T06:35:41.813862+0000","last_fullsized":"2024-04-22T06:35:41.813862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811335+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:35:41.811335+0000","last_peered":"2024-04-22T06:35:41.811335+0000","last_clean":"2024-04-22T06:35:41.811335+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:35:41.811335+0000","last_undegraded":"2024-04-22T06:35:41.811335+0000","last_fullsized":"2024-04-22T06:35:41.811335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078581+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:35:42.078581+0000","last_peered":"2024-04-22T06:35:42.078581+0000","last_clean":"2024-04-22T06:35:42.078581+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:35:42.078581+0000","last_undegraded":"2024-04-22T06:35:42.078581+0000","last_fullsized":"2024-04-22T06:35:42.078581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.033692+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:35:44.033692+0000","last_peered":"2024-04-22T06:35:44.033692+0000","last_clean":"2024-04-22T06:35:44.033692+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:35:44.033692+0000","last_undegraded":"2024-04-22T06:35:44.033692+0000","last_fullsized":"2024-04-22T06:35:44.033692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813361+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:35:41.813361+0000","last_peered":"2024-04-22T06:35:41.813361+0000","last_clean":"2024-04-22T06:35:41.813361+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:35:41.813361+0000","last_undegraded":"2024-04-22T06:35:41.813361+0000","last_fullsized":"2024-04-22T06:35:41.813361+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027345+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:35:43.027345+0000","last_peered":"2024-04-22T06:35:43.027345+0000","last_clean":"2024-04-22T06:35:43.027345+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:35:43.027345+0000","last_undegraded":"2024-04-22T06:35:43.027345+0000","last_fullsized":"2024-04-22T06:35:43.027345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810501+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:35:41.810501+0000","last_peered":"2024-04-22T06:35:41.810501+0000","last_clean":"2024-04-22T06:35:41.810501+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:35:41.810501+0000","last_undegraded":"2024-04-22T06:35:41.810501+0000","last_fullsized":"2024-04-22T06:35:41.810501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811795+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:35:41.811795+0000","last_peered":"2024-04-22T06:35:41.811795+0000","last_clean":"2024-04-22T06:35:41.811795+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:35:41.811795+0000","last_undegraded":"2024-04-22T06:35:41.811795+0000","last_fullsized":"2024-04-22T06:35:41.811795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.825426+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:35:40.825426+0000","last_peered":"2024-04-22T06:35:40.825426+0000","last_clean":"2024-04-22T06:35:40.825426+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:35:40.825426+0000","last_undegraded":"2024-04-22T06:35:40.825426+0000","last_fullsized":"2024-04-22T06:35:40.825426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:59:18.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027538+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:35:43.027538+0000","last_peered":"2024-04-22T06:35:43.027538+0000","last_clean":"2024-04-22T06:35:43.027538+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:35:43.027538+0000","last_undegraded":"2024-04-22T06:35:43.027538+0000","last_fullsized":"2024-04-22T06:35:43.027538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077847+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:35:42.077847+0000","last_peered":"2024-04-22T06:35:42.077847+0000","last_clean":"2024-04-22T06:35:42.077847+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:35:42.077847+0000","last_undegraded":"2024-04-22T06:35:42.077847+0000","last_fullsized":"2024-04-22T06:35:42.077847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.025736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:35:43.025736+0000","last_peered":"2024-04-22T06:35:43.025736+0000","last_clean":"2024-04-22T06:35:43.025736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:35:43.025736+0000","last_undegraded":"2024-04-22T06:35:43.025736+0000","last_fullsized":"2024-04-22T06:35:43.025736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820945+0000","last_change":"2024-04-22T06:35:40.817278+0000","last_active":"2024-04-22T06:35:40.820945+0000","last_peered":"2024-04-22T06:35:40.820945+0000","last_clean":"2024-04-22T06:35:40.820945+0000","last_became_active":"2024-04-22T06:35:40.816931+0000","last_became_peered":"2024-04-22T06:35:40.816931+0000","last_unstale":"2024-04-22T06:35:40.820945+0000","last_undegraded":"2024-04-22T06:35:40.820945+0000","last_fullsized":"2024-04-22T06:35:40.820945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:43:20.846914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811518+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:35:41.811518+0000","last_peered":"2024-04-22T06:35:41.811518+0000","last_clean":"2024-04-22T06:35:41.811518+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:35:41.811518+0000","last_undegraded":"2024-04-22T06:35:41.811518+0000","last_fullsized":"2024-04-22T06:35:41.811518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814031+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:35:41.814031+0000","last_peered":"2024-04-22T06:35:41.814031+0000","last_clean":"2024-04-22T06:35:41.814031+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:35:41.814031+0000","last_undegraded":"2024-04-22T06:35:41.814031+0000","last_fullsized":"2024-04-22T06:35:41.814031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.033836+0000","last_change":"2024-04-22T06:35:39.805809+0000","last_active":"2024-04-22T06:35:44.033836+0000","last_peered":"2024-04-22T06:35:44.033836+0000","last_clean":"2024-04-22T06:35:44.033836+0000","last_became_active":"2024-04-22T06:35:39.805308+0000","last_became_peered":"2024-04-22T06:35:39.805308+0000","last_unstale":"2024-04-22T06:35:44.033836+0000","last_undegraded":"2024-04-22T06:35:44.033836+0000","last_fullsized":"2024-04-22T06:35:44.033836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:00.335655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077982+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:35:42.077982+0000","last_peered":"2024-04-22T06:35:42.077982+0000","last_clean":"2024-04-22T06:35:42.077982+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:35:42.077982+0000","last_undegraded":"2024-04-22T06:35:42.077982+0000","last_fullsized":"2024-04-22T06:35:42.077982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027515+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:35:43.027515+0000","last_peered":"2024-04-22T06:35:43.027515+0000","last_clean":"2024-04-22T06:35:43.027515+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:35:43.027515+0000","last_undegraded":"2024-04-22T06:35:43.027515+0000","last_fullsized":"2024-04-22T06:35:43.027515+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947896+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:35:41.947896+0000","last_peered":"2024-04-22T06:35:41.947896+0000","last_clean":"2024-04-22T06:35:41.947896+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:35:41.947896+0000","last_undegraded":"2024-04-22T06:35:41.947896+0000","last_fullsized":"2024-04-22T06:35:41.947896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.043267+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:35:45.043267+0000","last_peered":"2024-04-22T06:35:45.043267+0000","last_clean":"2024-04-22T06:35:45.043267+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:35:45.043267+0000","last_undegraded":"2024-04-22T06:35:45.043267+0000","last_fullsized":"2024-04-22T06:35:45.043267+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947416+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:35:41.947416+0000","last_peered":"2024-04-22T06:35:41.947416+0000","last_clean":"2024-04-22T06:35:41.947416+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:35:41.947416+0000","last_undegraded":"2024-04-22T06:35:41.947416+0000","last_fullsized":"2024-04-22T06:35:41.947416+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813620+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:35:41.813620+0000","last_peered":"2024-04-22T06:35:41.813620+0000","last_clean":"2024-04-22T06:35:41.813620+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:35:41.813620+0000","last_undegraded":"2024-04-22T06:35:41.813620+0000","last_fullsized":"2024-04-22T06:35:41.813620+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078182+0000","last_change":"2024-04-22T06:35:40.819992+0000","last_active":"2024-04-22T06:35:42.078182+0000","last_peered":"2024-04-22T06:35:42.078182+0000","last_clean":"2024-04-22T06:35:42.078182+0000","last_became_active":"2024-04-22T06:35:40.819652+0000","last_became_peered":"2024-04-22T06:35:40.819652+0000","last_unstale":"2024-04-22T06:35:42.078182+0000","last_undegraded":"2024-04-22T06:35:42.078182+0000","last_fullsized":"2024-04-22T06:35:42.078182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:47:34.515492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814151+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:35:41.814151+0000","last_peered":"2024-04-22T06:35:41.814151+0000","last_clean":"2024-04-22T06:35:41.814151+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:35:41.814151+0000","last_undegraded":"2024-04-22T06:35:41.814151+0000","last_fullsized":"2024-04-22T06:35:41.814151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819751+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:35:40.819751+0000","last_peered":"2024-04-22T06:35:40.819751+0000","last_clean":"2024-04-22T06:35:40.819751+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:35:40.819751+0000","last_undegraded":"2024-04-22T06:35:40.819751+0000","last_fullsized":"2024-04-22T06:35:40.819751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:02.228001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.810642+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:35:41.810642+0000","last_peered":"2024-04-22T06:35:41.810642+0000","last_clean":"2024-04-22T06:35:41.810642+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:35:41.810642+0000","last_undegraded":"2024-04-22T06:35:41.810642+0000","last_fullsized":"2024-04-22T06:35:41.810642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811834+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:35:41.811834+0000","last_peered":"2024-04-22T06:35:41.811834+0000","last_clean":"2024-04-22T06:35:41.811834+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:35:41.811834+0000","last_undegraded":"2024-04-22T06:35:41.811834+0000","last_fullsized":"2024-04-22T06:35:41.811834+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947529+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:35:41.947529+0000","last_peered":"2024-04-22T06:35:41.947529+0000","last_clean":"2024-04-22T06:35:41.947529+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:35:41.947529+0000","last_undegraded":"2024-04-22T06:35:41.947529+0000","last_fullsized":"2024-04-22T06:35:41.947529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.811623+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:35:41.811623+0000","last_peered":"2024-04-22T06:35:41.811623+0000","last_clean":"2024-04-22T06:35:41.811623+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:35:41.811623+0000","last_undegraded":"2024-04-22T06:35:41.811623+0000","last_fullsized":"2024-04-22T06:35:41.811623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078205+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:35:42.078205+0000","last_peered":"2024-04-22T06:35:42.078205+0000","last_clean":"2024-04-22T06:35:42.078205+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:35:42.078205+0000","last_undegraded":"2024-04-22T06:35:42.078205+0000","last_fullsized":"2024-04-22T06:35:42.078205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.035142+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:35:44.035142+0000","last_peered":"2024-04-22T06:35:44.035142+0000","last_clean":"2024-04-22T06:35:44.035142+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:35:44.035142+0000","last_undegraded":"2024-04-22T06:35:44.035142+0000","last_fullsized":"2024-04-22T06:35:44.035142+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813419+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:35:41.813419+0000","last_peered":"2024-04-22T06:35:41.813419+0000","last_clean":"2024-04-22T06:35:41.813419+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:35:41.813419+0000","last_undegraded":"2024-04-22T06:35:41.813419+0000","last_fullsized":"2024-04-22T06:35:41.813419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.044907+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:35:45.044907+0000","last_peered":"2024-04-22T06:35:45.044907+0000","last_clean":"2024-04-22T06:35:45.044907+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:35:45.044907+0000","last_undegraded":"2024-04-22T06:35:45.044907+0000","last_fullsized":"2024-04-22T06:35:45.044907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947311+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:35:41.947311+0000","last_peered":"2024-04-22T06:35:41.947311+0000","last_clean":"2024-04-22T06:35:41.947311+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:35:41.947311+0000","last_undegraded":"2024-04-22T06:35:41.947311+0000","last_fullsized":"2024-04-22T06:35:41.947311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077266+0000","last_change":"2024-04-22T06:35:40.816889+0000","last_active":"2024-04-22T06:35:42.077266+0000","last_peered":"2024-04-22T06:35:42.077266+0000","last_clean":"2024-04-22T06:35:42.077266+0000","last_became_active":"2024-04-22T06:35:40.816555+0000","last_became_peered":"2024-04-22T06:35:40.816555+0000","last_unstale":"2024-04-22T06:35:42.077266+0000","last_undegraded":"2024-04-22T06:35:42.077266+0000","last_fullsized":"2024-04-22T06:35:42.077266+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:33:23.395675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.027170+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:35:43.027170+0000","last_peered":"2024-04-22T06:35:43.027170+0000","last_clean":"2024-04-22T06:35:43.027170+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:35:43.027170+0000","last_undegraded":"2024-04-22T06:35:43.027170+0000","last_fullsized":"2024-04-22T06:35:43.027170+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.814047+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:35:41.814047+0000","last_peered":"2024-04-22T06:35:41.814047+0000","last_clean":"2024-04-22T06:35:41.814047+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:35:41.814047+0000","last_undegraded":"2024-04-22T06:35:41.814047+0000","last_fullsized":"2024-04-22T06:35:41.814047+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078163+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:35:42.078163+0000","last_peered":"2024-04-22T06:35:42.078163+0000","last_clean":"2024-04-22T06:35:42.078163+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:35:42.078163+0000","last_undegraded":"2024-04-22T06:35:42.078163+0000","last_fullsized":"2024-04-22T06:35:42.078163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813414+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:35:41.813414+0000","last_peered":"2024-04-22T06:35:41.813414+0000","last_clean":"2024-04-22T06:35:41.813414+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:35:41.813414+0000","last_undegraded":"2024-04-22T06:35:41.813414+0000","last_fullsized":"2024-04-22T06:35:41.813414+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813655+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:35:41.813655+0000","last_peered":"2024-04-22T06:35:41.813655+0000","last_clean":"2024-04-22T06:35:41.813655+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:35:41.813655+0000","last_undegraded":"2024-04-22T06:35:41.813655+0000","last_fullsized":"2024-04-22T06:35:41.813655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.026959+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:35:43.026959+0000","last_peered":"2024-04-22T06:35:43.026959+0000","last_clean":"2024-04-22T06:35:43.026959+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:35:43.026959+0000","last_undegraded":"2024-04-22T06:35:43.026959+0000","last_fullsized":"2024-04-22T06:35:43.026959+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077854+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:35:42.077854+0000","last_peered":"2024-04-22T06:35:42.077854+0000","last_clean":"2024-04-22T06:35:42.077854+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:35:42.077854+0000","last_undegraded":"2024-04-22T06:35:42.077854+0000","last_fullsized":"2024-04-22T06:35:42.077854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.800396+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:35:40.800396+0000","last_peered":"2024-04-22T06:35:40.800396+0000","last_clean":"2024-04-22T06:35:40.800396+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:35:40.800396+0000","last_undegraded":"2024-04-22T06:35:40.800396+0000","last_fullsized":"2024-04-22T06:35:40.800396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813748+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:35:41.813748+0000","last_peered":"2024-04-22T06:35:41.813748+0000","last_clean":"2024-04-22T06:35:41.813748+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:35:41.813748+0000","last_undegraded":"2024-04-22T06:35:41.813748+0000","last_fullsized":"2024-04-22T06:35:41.813748+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:43.026069+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:35:43.026069+0000","last_peered":"2024-04-22T06:35:43.026069+0000","last_clean":"2024-04-22T06:35:43.026069+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:35:43.026069+0000","last_undegraded":"2024-04-22T06:35:43.026069+0000","last_fullsized":"2024-04-22T06:35:43.026069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.806899+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:35:40.806899+0000","last_peered":"2024-04-22T06:35:40.806899+0000","last_clean":"2024-04-22T06:35:40.806899+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:35:40.806899+0000","last_undegraded":"2024-04-22T06:35:40.806899+0000","last_fullsized":"2024-04-22T06:35:40.806899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:28:46.327532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:44.021959+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:35:44.021959+0000","last_peered":"2024-04-22T06:35:44.021959+0000","last_clean":"2024-04-22T06:35:44.021959+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:35:44.021959+0000","last_undegraded":"2024-04-22T06:35:44.021959+0000","last_fullsized":"2024-04-22T06:35:44.021959+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813291+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:35:41.813291+0000","last_peered":"2024-04-22T06:35:41.813291+0000","last_clean":"2024-04-22T06:35:41.813291+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:35:41.813291+0000","last_undegraded":"2024-04-22T06:35:41.813291+0000","last_fullsized":"2024-04-22T06:35:41.813291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813706+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:35:41.813706+0000","last_peered":"2024-04-22T06:35:41.813706+0000","last_clean":"2024-04-22T06:35:41.813706+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:35:41.813706+0000","last_undegraded":"2024-04-22T06:35:41.813706+0000","last_fullsized":"2024-04-22T06:35:41.813706+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947721+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:35:41.947721+0000","last_peered":"2024-04-22T06:35:41.947721+0000","last_clean":"2024-04-22T06:35:41.947721+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:35:41.947721+0000","last_undegraded":"2024-04-22T06:35:41.947721+0000","last_fullsized":"2024-04-22T06:35:41.947721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:45.045173+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:35:45.045173+0000","last_peered":"2024-04-22T06:35:45.045173+0000","last_clean":"2024-04-22T06:35:45.045173+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:35:45.045173+0000","last_undegraded":"2024-04-22T06:35:45.045173+0000","last_fullsized":"2024-04-22T06:35:45.045173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947921+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:35:41.947921+0000","last_peered":"2024-04-22T06:35:41.947921+0000","last_clean":"2024-04-22T06:35:41.947921+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:35:41.947921+0000","last_undegraded":"2024-04-22T06:35:41.947921+0000","last_fullsized":"2024-04-22T06:35:41.947921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"88'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078260+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:35:42.078260+0000","last_peered":"2024-04-22T06:35:42.078260+0000","last_clean":"2024-04-22T06:35:42.078260+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:35:42.078260+0000","last_undegraded":"2024-04-22T06:35:42.078260+0000","last_fullsized":"2024-04-22T06:35:42.078260+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947766+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:35:41.947766+0000","last_peered":"2024-04-22T06:35:41.947766+0000","last_clean":"2024-04-22T06:35:41.947766+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:35:41.947766+0000","last_undegraded":"2024-04-22T06:35:41.947766+0000","last_fullsized":"2024-04-22T06:35:41.947766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078116+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:35:42.078116+0000","last_peered":"2024-04-22T06:35:42.078116+0000","last_clean":"2024-04-22T06:35:42.078116+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:35:42.078116+0000","last_undegraded":"2024-04-22T06:35:42.078116+0000","last_fullsized":"2024-04-22T06:35:42.078116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.947838+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:35:41.947838+0000","last_peered":"2024-04-22T06:35:41.947838+0000","last_clean":"2024-04-22T06:35:41.947838+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:35:41.947838+0000","last_undegraded":"2024-04-22T06:35:41.947838+0000","last_fullsized":"2024-04-22T06:35:41.947838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.819755+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:35:40.819755+0000","last_peered":"2024-04-22T06:35:40.819755+0000","last_clean":"2024-04-22T06:35:40.819755+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:35:40.819755+0000","last_undegraded":"2024-04-22T06:35:40.819755+0000","last_fullsized":"2024-04-22T06:35:40.819755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:18:52.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812553+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:35:41.812553+0000","last_peered":"2024-04-22T06:35:41.812553+0000","last_clean":"2024-04-22T06:35:41.812553+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:35:41.812553+0000","last_undegraded":"2024-04-22T06:35:41.812553+0000","last_fullsized":"2024-04-22T06:35:41.812553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812235+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:35:41.812235+0000","last_peered":"2024-04-22T06:35:41.812235+0000","last_clean":"2024-04-22T06:35:41.812235+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:35:41.812235+0000","last_undegraded":"2024-04-22T06:35:41.812235+0000","last_fullsized":"2024-04-22T06:35:41.812235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077557+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:35:42.077557+0000","last_peered":"2024-04-22T06:35:42.077557+0000","last_clean":"2024-04-22T06:35:42.077557+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:35:42.077557+0000","last_undegraded":"2024-04-22T06:35:42.077557+0000","last_fullsized":"2024-04-22T06:35:42.077557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078299+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:35:42.078299+0000","last_peered":"2024-04-22T06:35:42.078299+0000","last_clean":"2024-04-22T06:35:42.078299+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:35:42.078299+0000","last_undegraded":"2024-04-22T06:35:42.078299+0000","last_fullsized":"2024-04-22T06:35:42.078299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.812166+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:35:40.812166+0000","last_peered":"2024-04-22T06:35:40.812166+0000","last_clean":"2024-04-22T06:35:40.812166+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:35:40.812166+0000","last_undegraded":"2024-04-22T06:35:40.812166+0000","last_fullsized":"2024-04-22T06:35:40.812166+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:09:26.679625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.812853+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:35:41.812853+0000","last_peered":"2024-04-22T06:35:41.812853+0000","last_clean":"2024-04-22T06:35:41.812853+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:35:41.812853+0000","last_undegraded":"2024-04-22T06:35:41.812853+0000","last_fullsized":"2024-04-22T06:35:41.812853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813655+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:35:41.813655+0000","last_peered":"2024-04-22T06:35:41.813655+0000","last_clean":"2024-04-22T06:35:41.813655+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:35:41.813655+0000","last_undegraded":"2024-04-22T06:35:41.813655+0000","last_fullsized":"2024-04-22T06:35:41.813655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.077996+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:35:42.077996+0000","last_peered":"2024-04-22T06:35:42.077996+0000","last_clean":"2024-04-22T06:35:42.077996+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:35:42.077996+0000","last_undegraded":"2024-04-22T06:35:42.077996+0000","last_fullsized":"2024-04-22T06:35:42.077996+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:41.813198+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:35:41.813198+0000","last_peered":"2024-04-22T06:35:41.813198+0000","last_clean":"2024-04-22T06:35:41.813198+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:35:41.813198+0000","last_undegraded":"2024-04-22T06:35:41.813198+0000","last_fullsized":"2024-04-22T06:35:41.813198+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.820902+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:35:40.820902+0000","last_peered":"2024-04-22T06:35:40.820902+0000","last_clean":"2024-04-22T06:35:40.820902+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:35:40.820902+0000","last_undegraded":"2024-04-22T06:35:40.820902+0000","last_fullsized":"2024-04-22T06:35:40.820902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:54:44.395476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-22T06:35:42.078171+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:35:42.078171+0000","last_peered":"2024-04-22T06:35:42.078171+0000","last_clean":"2024-04-22T06:35:42.078171+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:35:42.078171+0000","last_undegraded":"2024-04-22T06:35:42.078171+0000","last_fullsized":"2024-04-22T06:35:42.078171+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-22T06:35:40.809169+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:35:40.809169+0000","last_peered":"2024-04-22T06:35:40.809169+0000","last_clean":"2024-04-22T06:35:40.809169+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:35:40.809169+0000","last_undegraded":"2024-04-22T06:35:40.809169+0000","last_fullsized":"2024-04-22T06:35:40.809169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:31:02.737060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5642,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":12804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":449941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514514,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36884,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708268,"statfs":{"total":95995035648,"available":95957266432,"internally_reserved":0,"allocated":1576960,"data_stored":1050685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710744,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":962560,"data_stored":459093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972380,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":966656,"data_stored":459115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266721,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1564672,"data_stored":1050083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430247,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":458601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":958464,"data_stored":451553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":459205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":461473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":458705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888112,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":459155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691882,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1581056,"data_stored":1050355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:35:48.080 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-22T06:35:48.080 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T06:35:48.080 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T06:35:48.348 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:48.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:48 smithi105 ceph-mon[27916]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:48.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:48 smithi105 ceph-mon[27916]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:35:48.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2022659661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/515863208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:48 smithi105 ceph-mon[27916]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:48 smithi112 ceph-mon[31236]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:48 smithi112 ceph-mon[31236]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:35:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2022659661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/515863208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:48 smithi112 ceph-mon[31236]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:48 smithi177 ceph-mon[31346]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:35:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:48 smithi177 ceph-mon[31346]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-22T06:35:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2022659661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/515863208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:48 smithi177 ceph-mon[31346]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:49.384 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:49 smithi105 ceph-mon[27916]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:49.384 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:49 smithi105 ceph-mon[27916]: pgmap v291: 97 pgs: 97 active+clean; 619 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-22T06:35:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:49 smithi112 ceph-mon[31236]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:49.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:49 smithi112 ceph-mon[31236]: pgmap v291: 97 pgs: 97 active+clean; 619 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-22T06:35:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:49 smithi177 ceph-mon[31346]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:35:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:49 smithi177 ceph-mon[31346]: pgmap v291: 97 pgs: 97 active+clean; 619 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 -- 172.21.15.105:0/2746597228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 msgr2=0x7fcd44101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 --2- 172.21.15.105:0/2746597228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd44101de0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd400099c0 tx=0x7fcd40009cd0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 -- 172.21.15.105:0/2746597228 shutdown_connections 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 --2- 172.21.15.105:0/2746597228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 0x7fcd44109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 --2- 172.21.15.105:0/2746597228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd44101de0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 --2- 172.21.15.105:0/2746597228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 0x7fcd44103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 -- 172.21.15.105:0/2746597228 >> 172.21.15.105:0/2746597228 conn(0x7fcd440fcf60 msgr2=0x7fcd440ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 -- 172.21.15.105:0/2746597228 shutdown_connections 2024-04-22T06:35:49.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.933+0000 7fcd4be92700 1 -- 172.21.15.105:0/2746597228 wait complete. 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 Processor -- start 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 -- start start 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 0x7fcd4419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 0x7fcd4419ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d4b0 con 0x7fcd441019c0 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d330 con 0x7fcd44102bc0 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4be92700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d630 con 0x7fcd44103560 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50024/0 (socket says 172.21.15.105:50024) 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 -- 172.21.15.105:0/147844172 learned_addr learned my addr 172.21.15.105:0/147844172 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4942d700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 0x7fcd4419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd4a42f700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 0x7fcd4419ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 -- 172.21.15.105:0/147844172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 msgr2=0x7fcd4419ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:49.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 0x7fcd4419ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 -- 172.21.15.105:0/147844172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 msgr2=0x7fcd4419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 0x7fcd4419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.934+0000 7fcd49c2e700 1 -- 172.21.15.105:0/147844172 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd441a0800 con 0x7fcd441019c0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd49c2e700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009f50 tx=0x7fcd400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40027070 con 0x7fcd441019c0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd40022470 con 0x7fcd441019c0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd441a0a90 con 0x7fcd441019c0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000f6c0 con 0x7fcd441019c0 2024-04-22T06:35:49.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.935+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd441a0f70 con 0x7fcd441019c0 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.936+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd44050f80 con 0x7fcd441019c0 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.937+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fcd4002a070 con 0x7fcd441019c0 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.938+0000 7fcd3affd700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 0x7fcd300a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.938+0000 7fcd4942d700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 0x7fcd300a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.938+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fcd400d1320 con 0x7fcd441019c0 2024-04-22T06:35:49.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.938+0000 7fcd4942d700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 0x7fcd300a6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd4406b800 tx=0x7fcd3400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:49.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:49.942+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd4005eff0 con 0x7fcd441019c0 2024-04-22T06:35:50.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.230+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fcd44063650 con 0x7fcd441019c0 2024-04-22T06:35:50.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.232+0000 7fcd3affd700 1 -- 172.21.15.105:0/147844172 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7fcd40013070 con 0x7fcd441019c0 2024-04-22T06:35:50.231 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2024-04-22T06:35:50.231 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBGBSZm9vfSDRAAyZ9mJslDV0IHX0Fc9TfETg== 2024-04-22T06:35:50.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.234+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 msgr2=0x7fcd300a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:50.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.234+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 0x7fcd300a6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd4406b800 tx=0x7fcd3400a300 comp rx=0 tx=0).stop 2024-04-22T06:35:50.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.234+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 msgr2=0x7fcd4406b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:50.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.234+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009f50 tx=0x7fcd400047b0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 shutdown_connections 2024-04-22T06:35:50.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcd300a3d80 0x7fcd300a6240 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd44103560 0x7fcd4419ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd441019c0 0x7fcd4406b280 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 --2- 172.21.15.105:0/147844172 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcd44102bc0 0x7fcd4419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:50.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 >> 172.21.15.105:0/147844172 conn(0x7fcd440fcf60 msgr2=0x7fcd44105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:50.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 shutdown_connections 2024-04-22T06:35:50.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:50.235+0000 7fcd4be92700 1 -- 172.21.15.105:0/147844172 wait complete. 2024-04-22T06:35:50.516 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:35:50.516 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:35:50.676 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:35:50.676 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:35:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T06:35:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/147844172' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T06:35:50.878 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:35:50.878 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T06:35:50.878 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:35:50.915 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:35:50.915 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:35:50.916 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T06:35:50.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:35:50.677+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-22T06:35:50.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:35:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:35:50.677+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-22T06:35:51.199 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:51.604 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:51 smithi105 ceph-mon[27916]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-22T06:35:51.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:51 smithi112 ceph-mon[31236]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-22T06:35:51.742 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:51 smithi177 ceph-mon[31346]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 11 KiB/s wr, 29 op/s 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 -- 172.21.15.105:0/2056212325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818108ab0 msgr2=0x7fb818108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/2056212325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818108ab0 0x7fb818108e90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb814009990 tx=0x7fb814009ca0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 -- 172.21.15.105:0/2056212325 shutdown_connections 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/2056212325 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818103480 0x7fb81810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/2056212325 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818108ab0 0x7fb818108e90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/2056212325 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818102ac0 0x7fb818102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.847+0000 7fb81f9d6700 1 -- 172.21.15.105:0/2056212325 >> 172.21.15.105:0/2056212325 conn(0x7fb818079bf0 msgr2=0x7fb81807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- 172.21.15.105:0/2056212325 shutdown_connections 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- 172.21.15.105:0/2056212325 wait complete. 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 Processor -- start 2024-04-22T06:35:52.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- start start 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818102ac0 0x7fb8181991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818108ab0 0x7fb81819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb818112cb0 con 0x7fb818103480 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb818112b30 con 0x7fb818108ab0 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.848+0000 7fb81f9d6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb818112e30 con 0x7fb818102ac0 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50044/0 (socket says 172.21.15.105:50044) 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 -- 172.21.15.105:0/1620381350 learned_addr learned my addr 172.21.15.105:0/1620381350 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81d772700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818102ac0 0x7fb8181991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:52.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81df73700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818108ab0 0x7fb81819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 -- 172.21.15.105:0/1620381350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818102ac0 msgr2=0x7fb8181991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818102ac0 0x7fb8181991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 -- 172.21.15.105:0/1620381350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818108ab0 msgr2=0x7fb81819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818108ab0 0x7fb81819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 -- 172.21.15.105:0/1620381350 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81819e1f0 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81cf71700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c920 tx=0x7fb80800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808007b00 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb808004ce0 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81819e4e0 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808005550 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.849+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb818074c70 con 0x7fb818103480 2024-04-22T06:35:52.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.850+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8181041f0 con 0x7fb818103480 2024-04-22T06:35:52.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.851+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb808007ca0 con 0x7fb818103480 2024-04-22T06:35:52.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.852+0000 7fb80e7fc700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 0x7fb8040a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:52.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.852+0000 7fb81d772700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 0x7fb8040a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:52.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.852+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fb8080d02e0 con 0x7fb818103480 2024-04-22T06:35:52.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.853+0000 7fb81d772700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 0x7fb8040a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb81400b5c0 tx=0x7fb814005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:52.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:52.857+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb808014030 con 0x7fb818103480 2024-04-22T06:35:53.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.113+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb81802df80 con 0x7fb818103480 2024-04-22T06:35:53.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.118+0000 7fb80e7fc700 1 -- 172.21.15.105:0/1620381350 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+265005 (secure 0 0 0) 0x7fb80809cd10 con 0x7fb818103480 2024-04-22T06:35:53.120 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 msgr2=0x7fb8040a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 0x7fb8040a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb81400b5c0 tx=0x7fb814005960 comp rx=0 tx=0).stop 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 msgr2=0x7fb818199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c920 tx=0x7fb80800cc30 comp rx=0 tx=0).stop 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 shutdown_connections 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb8040a3cd0 0x7fb8040a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb818108ab0 0x7fb81819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb818102ac0 0x7fb8181991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 --2- 172.21.15.105:0/1620381350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb818103480 0x7fb818199720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:53.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 >> 172.21.15.105:0/1620381350 conn(0x7fb818079bf0 msgr2=0x7fb818100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:53.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 shutdown_connections 2024-04-22T06:35:53.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:53.122+0000 7fb81f9d6700 1 -- 172.21.15.105:0/1620381350 wait complete. 2024-04-22T06:35:53.787 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2556874834},{"type":"v1","addr":"172.21.15.105:6801","nonce":2556874834}]},"active_addr":"172.21.15.105:6801/2556874834","active_change":"2024-04-22T06:28:44.539384+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.105:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":817153020}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":3570614146}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1014082495}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":3300341732}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":804987608}]}]} 2024-04-22T06:35:53.793 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T06:35:53.793 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T06:35:53.793 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:35:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:53 smithi112 ceph-mon[31236]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-04-22T06:35:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:53 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1620381350' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:35:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:53 smithi177 ceph-mon[31346]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-04-22T06:35:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:53 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1620381350' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:35:54.029 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:54.133 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:53 smithi105 ceph-mon[27916]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 10 KiB/s wr, 28 op/s 2024-04-22T06:35:54.133 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:53 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1620381350' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T06:35:55.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 -- 172.21.15.105:0/812897358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4108a50 msgr2=0x7f2ff4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 --2- 172.21.15.105:0/812897358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4108a50 0x7f2ff4108e30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009990 tx=0x7f2fe4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 -- 172.21.15.105:0/812897358 shutdown_connections 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 --2- 172.21.15.105:0/812897358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4103450 0x7f2ff410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 --2- 172.21.15.105:0/812897358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4108a50 0x7f2ff4108e30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 --2- 172.21.15.105:0/812897358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 -- 172.21.15.105:0/812897358 >> 172.21.15.105:0/812897358 conn(0x7f2ff4079ba0 msgr2=0x7f2ff4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 -- 172.21.15.105:0/812897358 shutdown_connections 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 -- 172.21.15.105:0/812897358 wait complete. 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.511+0000 7f2ff9303700 1 Processor -- start 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 -- start start 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4108a50 0x7f2ff40757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112c60 con 0x7f2ff4103450 2024-04-22T06:35:55.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112ae0 con 0x7f2ff4102a90 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff9303700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff4112de0 con 0x7f2ff4108a50 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50678/0 (socket says 172.21.15.105:50678) 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 -- 172.21.15.105:0/3855456920 learned_addr learned my addr 172.21.15.105:0/3855456920 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff2ffd700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff37fe700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4108a50 0x7f2ff40757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 -- 172.21.15.105:0/3855456920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4108a50 msgr2=0x7f2ff40757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4108a50 0x7f2ff40757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 -- 172.21.15.105:0/3855456920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 msgr2=0x7f2ff4074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 -- 172.21.15.105:0/3855456920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff41a76e0 con 0x7f2ff4103450 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2ff2ffd700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.512+0000 7f2fea7fc700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c920 tx=0x7f2fdc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.513+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007b00 con 0x7f2ff4103450 2024-04-22T06:35:55.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.513+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fdc004ce0 con 0x7f2ff4103450 2024-04-22T06:35:55.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.513+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc005650 con 0x7f2ff4103450 2024-04-22T06:35:55.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.513+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff41a79d0 con 0x7f2ff4103450 2024-04-22T06:35:55.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.513+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff41a7ec0 con 0x7f2ff4103450 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.514+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff4069730 con 0x7f2ff4103450 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.515+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2fdc004750 con 0x7f2ff4103450 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.515+0000 7f2ff0ff9700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 0x7f2fe00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.515+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f2fdc0d0390 con 0x7f2ff4103450 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.515+0000 7f2ff2ffd700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 0x7f2fe00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:55.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.516+0000 7f2ff2ffd700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 0x7f2fe00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400b5c0 tx=0x7f2fe4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:55.517 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.519+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fdc014030 con 0x7f2ff4103450 2024-04-22T06:35:55.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.720+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ff402e020 con 0x7f2ff4103450 2024-04-22T06:35:55.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.721+0000 7f2ff0ff9700 1 -- 172.21.15.105:0/3855456920 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22265 (secure 0 0 0) 0x7f2fdc09cdc0 con 0x7f2ff4103450 2024-04-22T06:35:55.720 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":89,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:35:41.801359+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 msgr2=0x7f2fe00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 0x7f2fe00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400b5c0 tx=0x7f2fe4005fb0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 msgr2=0x7f2ff4075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c920 tx=0x7f2fdc00cce0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 shutdown_connections 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f2fe00a3cd0 0x7f2fe00a6190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ff4102a90 0x7f2ff4074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2ff4108a50 0x7f2ff40757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 --2- 172.21.15.105:0/3855456920 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ff4103450 0x7f2ff4075280 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:55.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.723+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 >> 172.21.15.105:0/3855456920 conn(0x7f2ff4079ba0 msgr2=0x7f2ff41018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:55.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.724+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 shutdown_connections 2024-04-22T06:35:55.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:55.724+0000 7f2ff9303700 1 -- 172.21.15.105:0/3855456920 wait complete. 2024-04-22T06:35:55.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:55 smithi105 ceph-mon[27916]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-04-22T06:35:55.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:55 smithi112 ceph-mon[31236]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-04-22T06:35:55.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:55 smithi177 ceph-mon[31346]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 KiB/s wr, 25 op/s 2024-04-22T06:35:56.348 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T06:35:56.348 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T06:35:56.583 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:56 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3855456920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:56.982 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:56 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3855456920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:56 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3855456920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:57.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:57 smithi105 ceph-mon[27916]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-04-22T06:35:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:57 smithi112 ceph-mon[31236]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-04-22T06:35:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:57 smithi177 ceph-mon[31346]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-04-22T06:35:58.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.149+0000 7fb1d1604700 1 -- 172.21.15.105:0/893470124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc06b140 msgr2=0x7fb1cc06b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.149+0000 7fb1d1604700 1 --2- 172.21.15.105:0/893470124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc06b140 0x7fb1cc06b5a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0009990 tx=0x7fb1c0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 -- 172.21.15.105:0/893470124 shutdown_connections 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 --2- 172.21.15.105:0/893470124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc108a60 0x7fb1cc108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 --2- 172.21.15.105:0/893470124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06bae0 0x7fb1cc10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 --2- 172.21.15.105:0/893470124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc06b140 0x7fb1cc06b5a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 -- 172.21.15.105:0/893470124 >> 172.21.15.105:0/893470124 conn(0x7fb1cc078980 msgr2=0x7fb1cc078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 -- 172.21.15.105:0/893470124 shutdown_connections 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 -- 172.21.15.105:0/893470124 wait complete. 2024-04-22T06:35:58.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 Processor -- start 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.150+0000 7fb1d1604700 1 -- start start 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc06bae0 0x7fb1cc1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 0x7fb1cc19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112c70 con 0x7fb1cc108a60 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112af0 con 0x7fb1cc06bae0 2024-04-22T06:35:58.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112df0 con 0x7fb1cc06b140 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:37568/0 (socket says 172.21.15.105:37568) 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1caffd700 1 -- 172.21.15.105:0/3018171546 learned_addr learned my addr 172.21.15.105:0/3018171546 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 0x7fb1cc19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1ca7fc700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc06bae0 0x7fb1cc1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 -- 172.21.15.105:0/3018171546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 msgr2=0x7fb1cc199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 -- 172.21.15.105:0/3018171546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc06bae0 msgr2=0x7fb1cc1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc06bae0 0x7fb1cc1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 -- 172.21.15.105:0/3018171546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc19e190 con 0x7fb1cc108a60 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1caffd700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 0x7fb1cc19da50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4007f10 tx=0x7fb1c400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:58.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4004db0 con 0x7fb1cc108a60 2024-04-22T06:35:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1cc19e480 con 0x7fb1cc108a60 2024-04-22T06:35:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c4003a00 con 0x7fb1cc108a60 2024-04-22T06:35:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4004580 con 0x7fb1cc108a60 2024-04-22T06:35:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.151+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1cc1a5d00 con 0x7fb1cc108a60 2024-04-22T06:35:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.152+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc1041a0 con 0x7fb1cc108a60 2024-04-22T06:35:58.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.154+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb1c401c020 con 0x7fb1cc108a60 2024-04-22T06:35:58.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.155+0000 7fb1b3fff700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 0x7fb1b40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:35:58.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.155+0000 7fb1caffd700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 0x7fb1b40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:35:58.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.155+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fb1c40d0590 con 0x7fb1cc108a60 2024-04-22T06:35:58.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.155+0000 7fb1caffd700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 0x7fb1b40a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0103d0 tx=0x7fb1bc007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:35:58.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.158+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1c4023030 con 0x7fb1cc108a60 2024-04-22T06:35:58.355 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.356+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1cc02df80 con 0x7fb1cc108a60 2024-04-22T06:35:58.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.358+0000 7fb1b3fff700 1 -- 172.21.15.105:0/3018171546 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22265 (secure 0 0 0) 0x7fb1c4010070 con 0x7fb1cc108a60 2024-04-22T06:35:58.357 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:35:58.357 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":89,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:35:41.801359+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 msgr2=0x7fb1b40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 0x7fb1b40a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0103d0 tx=0x7fb1bc007480 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 msgr2=0x7fb1cc19da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 0x7fb1cc19da50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4007f10 tx=0x7fb1c400d3b0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 shutdown_connections 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3cd0 0x7fb1b40a6190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc108a60 0x7fb1cc19da50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc06b140 0x7fb1cc199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 --2- 172.21.15.105:0/3018171546 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc06bae0 0x7fb1cc1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 >> 172.21.15.105:0/3018171546 conn(0x7fb1cc078980 msgr2=0x7fb1cc0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 shutdown_connections 2024-04-22T06:35:58.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:35:58.360+0000 7fb1d1604700 1 -- 172.21.15.105:0/3018171546 wait complete. 2024-04-22T06:35:58.936 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:58 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3018171546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:58.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T06:35:58.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T06:35:58.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T06:35:58.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T06:35:58.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T06:35:58.940 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T06:35:58.940 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T06:35:58.940 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T06:35:58.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-22T06:35:58.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-22T06:35:58.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-22T06:35:58.942 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-22T06:35:58.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:58 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3018171546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:58.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:58 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3018171546' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:35:59.667 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.667 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.670 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.670 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.670 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.671 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.671 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.674 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.679 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.679 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.684 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.698 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:35:59.949 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:35:59 smithi105 ceph-mon[27916]: pgmap v296: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-22T06:35:59.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:35:59 smithi112 ceph-mon[31236]: pgmap v296: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-22T06:35:59.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:35:59 smithi177 ceph-mon[31346]: pgmap v296: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-22T06:36:01.845 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:01 smithi105 ceph-mon[27916]: pgmap v297: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-22T06:36:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:01 smithi112 ceph-mon[31236]: pgmap v297: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-22T06:36:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:01 smithi177 ceph-mon[31346]: pgmap v297: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-04-22T06:36:03.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:03 smithi112 ceph-mon[31236]: pgmap v298: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-22T06:36:03.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:03 smithi177 ceph-mon[31346]: pgmap v298: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-22T06:36:04.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:03 smithi105 ceph-mon[27916]: pgmap v298: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-22T06:36:05.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:05 smithi112 ceph-mon[31236]: pgmap v299: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-22T06:36:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:05 smithi177 ceph-mon[31346]: pgmap v299: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-22T06:36:06.013 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:05 smithi105 ceph-mon[27916]: pgmap v299: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-22T06:36:07.850 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:07 smithi105 ceph-mon[27916]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-22T06:36:07.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:07 smithi112 ceph-mon[31236]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-22T06:36:07.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:07 smithi177 ceph-mon[31346]: pgmap v300: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 -- 172.21.15.105:0/619447842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b006bb10 msgr2=0x7f70b010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/619447842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b006bb10 0x7f70b010f900 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f70a800b0d0 tx=0x7f70a800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 -- 172.21.15.105:0/619447842 shutdown_connections 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/619447842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b006bb10 0x7f70b010f900 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/619447842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b0108a50 0x7f70b0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/619447842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70b006b150 0x7f70b006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 -- 172.21.15.105:0/619447842 >> 172.21.15.105:0/619447842 conn(0x7f70b0077990 msgr2=0x7f70b0077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 -- 172.21.15.105:0/619447842 shutdown_connections 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 -- 172.21.15.105:0/619447842 wait complete. 2024-04-22T06:36:09.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.735+0000 7f70b4a3f700 1 Processor -- start 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 -- start start 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b006b150 0x7f70b01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70b006bb10 0x7f70b01996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 0x7f70b019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0112c90 con 0x7f70b0108a50 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0112b10 con 0x7f70b006bb10 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70b4a3f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0112e10 con 0x7f70b006b150 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 0x7f70b019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 0x7f70b019da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41936/0 (socket says 172.21.15.105:41936) 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 -- 172.21.15.105:0/3562575573 learned_addr learned my addr 172.21.15.105:0/3562575573 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70ae59c700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b006b150 0x7f70b01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70a5d9b700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70b006bb10 0x7f70b01996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:09.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 -- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b006b150 msgr2=0x7f70b01991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f70b006b150 0x7f70b01991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 -- 172.21.15.105:0/3562575573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70b006bb10 msgr2=0x7f70b01996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f70b006bb10 0x7f70b01996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.736+0000 7f70aed9d700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b019e1b0 con 0x7f70b0108a50 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70aed9d700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 0x7f70b019da70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009fd0 tx=0x7f70a8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a801f070 con 0x7f70b0108a50 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a8003c00 con 0x7f70b0108a50 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b019e440 con 0x7f70b0108a50 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a801b7b0 con 0x7f70b0108a50 2024-04-22T06:36:09.739 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.737+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b01a5d20 con 0x7f70b0108a50 2024-04-22T06:36:09.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.738+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f70b00630d0 con 0x7f70b0108a50 2024-04-22T06:36:09.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.739+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f70a8022070 con 0x7f70b0108a50 2024-04-22T06:36:09.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70a7fff700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f709c0a3cd0 0x7f709c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:09.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70ae59c700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f709c0a3cd0 0x7f709c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f70a80d15b0 con 0x7f70b0108a50 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70a7fff700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] conn(0x7f709c0ad4b0 0x7f709c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70a8008ea0 con 0x7f709c0ad4b0 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f70a80d1a90 con 0x7f70b0108a50 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.740+0000 7f70ae59c700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f709c0a3cd0 0x7f709c0a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009a10 tx=0x7f70a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.741+0000 7f70a5d9b700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] conn(0x7f709c0ad4b0 0x7f709c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.741+0000 7f70a5d9b700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] conn(0x7f709c0ad4b0 0x7f709c0af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:09.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.742+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== osd.2 v2:172.21.15.177:6800/1615473709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70a8008ea0 con 0x7f709c0ad4b0 2024-04-22T06:36:09.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.758+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 --> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f70b0063550 con 0x7f709c0ad4b0 2024-04-22T06:36:09.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70a7fff700 1 -- 172.21.15.105:0/3562575573 <== osd.2 v2:172.21.15.177:6800/1615473709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f70b0063550 con 0x7f709c0ad4b0 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] conn(0x7f709c0ad4b0 msgr2=0x7f709c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] conn(0x7f709c0ad4b0 0x7f709c0af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f709c0a3cd0 msgr2=0x7f709c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f709c0a3cd0 0x7f709c0a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009a10 tx=0x7f70a0009190 comp rx=0 tx=0).stop 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 msgr2=0x7f70b019da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 --2- 172.21.15.105:0/3562575573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70b0108a50 0x7f70b019da70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f70a8009fd0 tx=0x7f70a8007a20 comp rx=0 tx=0).stop 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70aed9d700 1 -- 172.21.15.105:0/3562575573 reap_dead start 2024-04-22T06:36:09.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 shutdown_connections 2024-04-22T06:36:09.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 >> 172.21.15.105:0/3562575573 conn(0x7f70b0077990 msgr2=0x7f70b0100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:09.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 shutdown_connections 2024-04-22T06:36:09.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:09.759+0000 7f70b4a3f700 1 -- 172.21.15.105:0/3562575573 wait complete. 2024-04-22T06:36:09.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:09 smithi105 ceph-mon[27916]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T06:36:09.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:09 smithi112 ceph-mon[31236]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T06:36:09.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:09 smithi177 ceph-mon[31346]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-22T06:36:10.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 -- 172.21.15.105:0/2324754891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006afe0 msgr2=0x7f8db006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 --2- 172.21.15.105:0/2324754891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006afe0 0x7f8db006b440 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009990 tx=0x7f8da0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 -- 172.21.15.105:0/2324754891 shutdown_connections 2024-04-22T06:36:10.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 --2- 172.21.15.105:0/2324754891 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db006b980 0x7f8db010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 --2- 172.21.15.105:0/2324754891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 0x7f8db0108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 --2- 172.21.15.105:0/2324754891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006afe0 0x7f8db006b440 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 -- 172.21.15.105:0/2324754891 >> 172.21.15.105:0/2324754891 conn(0x7f8db00788b0 msgr2=0x7f8db0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 -- 172.21.15.105:0/2324754891 shutdown_connections 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.861+0000 7f8db7f87700 1 -- 172.21.15.105:0/2324754891 wait complete. 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 Processor -- start 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 -- start start 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db006afe0 0x7f8db01992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 0x7f8db01997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 0x7f8db019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8db0112ca0 con 0x7f8db006b980 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8db0112b20 con 0x7f8db006afe0 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db7f87700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8db0112e20 con 0x7f8db0108a20 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db5522700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 0x7f8db01997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:10.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db5522700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 0x7f8db01997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41966/0 (socket says 172.21.15.105:41966) 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db5522700 1 -- 172.21.15.105:0/1846964455 learned_addr learned my addr 172.21.15.105:0/1846964455 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.862+0000 7f8db5d23700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db006afe0 0x7f8db01992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db6524700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 0x7f8db019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 -- 172.21.15.105:0/1846964455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 msgr2=0x7f8db019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 0x7f8db019db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 -- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db006afe0 msgr2=0x7f8db01992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8db006afe0 0x7f8db01992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db019e2b0 con 0x7f8db006b980 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db6524700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8db0108a20 0x7f8db019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db5522700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 0x7f8db01997e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8da00057f0 tx=0x7f8da0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:10.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0027070 con 0x7f8db006b980 2024-04-22T06:36:10.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da00040a0 con 0x7f8db006b980 2024-04-22T06:36:10.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db019e540 con 0x7f8db006b980 2024-04-22T06:36:10.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0022dc0 con 0x7f8db006b980 2024-04-22T06:36:10.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.863+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db01a5e20 con 0x7f8db006b980 2024-04-22T06:36:10.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.864+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8db00631d0 con 0x7f8db006b980 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.866+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8da002a070 con 0x7f8db006b980 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.866+0000 7f8da6ffd700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d9c0a3d50 0x7f8d9c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.866+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f8da00d1f70 con 0x7f8db006b980 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.866+0000 7f8db5d23700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d9c0a3d50 0x7f8d9c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8da6ffd700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] conn(0x7f8d9c0ad530 0x7f8d9c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8da0009030 con 0x7f8d9c0ad530 2024-04-22T06:36:10.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f8da00d2360 con 0x7f8db006b980 2024-04-22T06:36:10.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8db5d23700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d9c0a3d50 0x7f8d9c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009a10 tx=0x7f8dac009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:10.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8db6524700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] conn(0x7f8d9c0ad530 0x7f8d9c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:10.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.867+0000 7f8db6524700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] conn(0x7f8d9c0ad530 0x7f8d9c0af950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:10.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.868+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== osd.10 v2:172.21.15.112:6824/2010680205 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8da0009030 con 0x7f8d9c0ad530 2024-04-22T06:36:10.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.886+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 --> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8db0063650 con 0x7f8d9c0ad530 2024-04-22T06:36:10.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.886+0000 7f8da6ffd700 1 -- 172.21.15.105:0/1846964455 <== osd.10 v2:172.21.15.112:6824/2010680205 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8db0063650 con 0x7f8d9c0ad530 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] conn(0x7f8d9c0ad530 msgr2=0x7f8d9c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] conn(0x7f8d9c0ad530 0x7f8d9c0af950 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d9c0a3d50 msgr2=0x7f8d9c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d9c0a3d50 0x7f8d9c0a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009a10 tx=0x7f8dac009190 comp rx=0 tx=0).stop 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 msgr2=0x7f8db01997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:10.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 --2- 172.21.15.105:0/1846964455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8db006b980 0x7f8db01997e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8da00057f0 tx=0x7f8da0003680 comp rx=0 tx=0).stop 2024-04-22T06:36:10.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db6524700 1 -- 172.21.15.105:0/1846964455 reap_dead start 2024-04-22T06:36:10.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 shutdown_connections 2024-04-22T06:36:10.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.887+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 >> 172.21.15.105:0/1846964455 conn(0x7f8db00788b0 msgr2=0x7f8db00ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:10.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.888+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 shutdown_connections 2024-04-22T06:36:10.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.888+0000 7f8db7f87700 1 -- 172.21.15.105:0/1846964455 wait complete. 2024-04-22T06:36:11.940 INFO:teuthology.orchestra.run.smithi105.stdout:77309411391 2024-04-22T06:36:11.940 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T06:36:11.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:11 smithi112 ceph-mon[31236]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:11.963 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:11 smithi105 ceph-mon[27916]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:11.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:11 smithi177 ceph-mon[31346]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:13.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 -- 172.21.15.105:0/1399949910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068107920 msgr2=0x7f5068107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 --2- 172.21.15.105:0/1399949910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068107920 0x7f5068107d00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5058009990 tx=0x7f5058009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 -- 172.21.15.105:0/1399949910 shutdown_connections 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 --2- 172.21.15.105:0/1399949910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5068102330 0x7f506810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 --2- 172.21.15.105:0/1399949910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068107920 0x7f5068107d00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 --2- 172.21.15.105:0/1399949910 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068101970 0x7f5068101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 -- 172.21.15.105:0/1399949910 >> 172.21.15.105:0/1399949910 conn(0x7f50680779b0 msgr2=0x7f5068077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 -- 172.21.15.105:0/1399949910 shutdown_connections 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.472+0000 7f506d9cf700 1 -- 172.21.15.105:0/1399949910 wait complete. 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 Processor -- start 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 -- start start 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 0x7f5068198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102330 0x7f5068198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:13.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5068107920 0x7f506819c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5068111bc0 con 0x7f5068101970 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5068111a40 con 0x7f5068107920 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.473+0000 7f506d9cf700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5068111d40 con 0x7f5068102330 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 0x7f5068198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 0x7f5068198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41952/0 (socket says 172.21.15.105:41952) 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 -- 172.21.15.105:0/3249616321 learned_addr learned my addr 172.21.15.105:0/3249616321 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f50667fc700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102330 0x7f5068198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 -- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102330 msgr2=0x7f5068198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5068102330 0x7f5068198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f50677fe700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5068107920 0x7f506819c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 -- 172.21.15.105:0/3249616321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5068107920 msgr2=0x7f506819c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5068107920 0x7f506819c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506819d120 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f5066ffd700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 0x7f5068198110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5058000c00 tx=0x7f50580047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5058027070 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50580041d0 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506819d3b0 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505800f460 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.474+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f506806b1b0 con 0x7f5068101970 2024-04-22T06:36:13.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.475+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f506806b760 con 0x7f5068101970 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.476+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f505802a070 con 0x7f5068101970 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f506c9cd700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f50500a3cd0 0x7f50500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f50667fc700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f50500a3cd0 0x7f50500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f50580d1ab0 con 0x7f5068101970 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f506c9cd700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] conn(0x7f50500ad4b0 0x7f50500af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50580043d0 con 0x7f50500ad4b0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f50580d1f90 con 0x7f5068101970 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f50667fc700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f50500a3cd0 0x7f50500a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.477+0000 7f50677fe700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] conn(0x7f50500ad4b0 0x7f50500af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.478+0000 7f50677fe700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] conn(0x7f50500ad4b0 0x7f50500af8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.479+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== osd.8 v2:172.21.15.177:6816/76921589 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50580043d0 con 0x7f50500ad4b0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.495+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 --> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f506806bbe0 con 0x7f50500ad4b0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.495+0000 7f506c9cd700 1 -- 172.21.15.105:0/3249616321 <== osd.8 v2:172.21.15.177:6816/76921589 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f506806bbe0 con 0x7f50500ad4b0 2024-04-22T06:36:13.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] conn(0x7f50500ad4b0 msgr2=0x7f50500af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] conn(0x7f50500ad4b0 0x7f50500af8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f50500a3cd0 msgr2=0x7f50500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f50500a3cd0 0x7f50500a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c008040 comp rx=0 tx=0).stop 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 msgr2=0x7f5068198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 --2- 172.21.15.105:0/3249616321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5068101970 0x7f5068198110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5058000c00 tx=0x7f50580047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f50677fe700 1 -- 172.21.15.105:0/3249616321 reap_dead start 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 shutdown_connections 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 >> 172.21.15.105:0/3249616321 conn(0x7f50680779b0 msgr2=0x7f50680ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 shutdown_connections 2024-04-22T06:36:13.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:10.496+0000 7f506d9cf700 1 -- 172.21.15.105:0/3249616321 wait complete. 2024-04-22T06:36:13.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:13 smithi105 ceph-mon[27916]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:13.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:13 smithi177 ceph-mon[31346]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:14.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:13 smithi112 ceph-mon[31236]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:14.319 INFO:teuthology.orchestra.run.smithi105.stdout:266287972386 2024-04-22T06:36:14.320 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-22T06:36:15.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 -- 172.21.15.105:0/207876231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 msgr2=0x7f8fac101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/207876231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac101dc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009990 tx=0x7f8f9c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 -- 172.21.15.105:0/207876231 shutdown_connections 2024-04-22T06:36:15.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/207876231 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fac107990 0x7f8fac107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/207876231 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac102300 0x7f8fac10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/207876231 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac101dc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 -- 172.21.15.105:0/207876231 >> 172.21.15.105:0/207876231 conn(0x7f8fac0788f0 msgr2=0x7f8fac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 -- 172.21.15.105:0/207876231 shutdown_connections 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.323+0000 7f8fb24ca700 1 -- 172.21.15.105:0/207876231 wait complete. 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 Processor -- start 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 -- start start 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fac102300 0x7f8fac1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac107990 0x7f8fac19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac111b50 con 0x7f8fac101940 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1119d0 con 0x7f8fac102300 2024-04-22T06:36:15.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fb24ca700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac111cd0 con 0x7f8fac107990 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fabfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fabfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42892/0 (socket says 172.21.15.105:42892) 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.324+0000 7f8fabfff700 1 -- 172.21.15.105:0/1601676698 learned_addr learned my addr 172.21.15.105:0/1601676698 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fab7fe700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fac102300 0x7f8fac1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fb0a67700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac107990 0x7f8fac19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 -- 172.21.15.105:0/1601676698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac107990 msgr2=0x7f8fac19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac107990 0x7f8fac19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 -- 172.21.15.105:0/1601676698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fac102300 msgr2=0x7f8fac1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fac102300 0x7f8fac1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac19d070 con 0x7f8fac101940 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fb0a67700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8fac107990 0x7f8fac19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:15.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fabfff700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac198060 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000d980 tx=0x7f8fa000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0016070 con 0x7f8fac101940 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac19d360 con 0x7f8fac101940 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa0004830 con 0x7f8fac101940 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00054c0 con 0x7f8fac101940 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.325+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1a4be0 con 0x7f8fac101940 2024-04-22T06:36:15.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.326+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8fac063150 con 0x7f8fac101940 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.327+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8fa0004220 con 0x7f8fac101940 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fa97fa700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8f940a3cd0 0x7f8f940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fab7fe700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8f940a3cd0 0x7f8f940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f8fa00d0890 con 0x7f8fac101940 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fa97fa700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] conn(0x7f8f940ad4b0 0x7f8f940af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8fa0015ec0 con 0x7f8f940ad4b0 2024-04-22T06:36:15.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f8fa00d0d70 con 0x7f8fac101940 2024-04-22T06:36:15.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fb0a67700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] conn(0x7f8f940ad4b0 0x7f8f940af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fab7fe700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8f940a3cd0 0x7f8f940a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00b5c0 tx=0x7f8f9c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.328+0000 7f8fb0a67700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] conn(0x7f8f940ad4b0 0x7f8f940af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.329+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== osd.0 v2:172.21.15.105:6802/1950705877 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8fa0015ec0 con 0x7f8f940ad4b0 2024-04-22T06:36:15.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.345+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 --> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8fac063580 con 0x7f8f940ad4b0 2024-04-22T06:36:15.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.345+0000 7f8fa97fa700 1 -- 172.21.15.105:0/1601676698 <== osd.0 v2:172.21.15.105:6802/1950705877 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8fac063580 con 0x7f8f940ad4b0 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] conn(0x7f8f940ad4b0 msgr2=0x7f8f940af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] conn(0x7f8f940ad4b0 0x7f8f940af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8f940a3cd0 msgr2=0x7f8f940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8f940a3cd0 0x7f8f940a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00b5c0 tx=0x7f8f9c005960 comp rx=0 tx=0).stop 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 msgr2=0x7f8fac198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 --2- 172.21.15.105:0/1601676698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fac101940 0x7f8fac198060 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8fa000d980 tx=0x7f8fa000dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb0a67700 1 -- 172.21.15.105:0/1601676698 reap_dead start 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 shutdown_connections 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 >> 172.21.15.105:0/1601676698 conn(0x7f8fac0788f0 msgr2=0x7f8fac0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 shutdown_connections 2024-04-22T06:36:15.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.346+0000 7f8fb24ca700 1 -- 172.21.15.105:0/1601676698 wait complete. 2024-04-22T06:36:15.855 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:15 smithi105 ceph-mon[27916]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:36:15.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 -- 172.21.15.105:0/464798164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c108a50 msgr2=0x7f968c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 --2- 172.21.15.105:0/464798164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c108a50 0x7f968c108e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f968800b0a0 tx=0x7f968800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 -- 172.21.15.105:0/464798164 shutdown_connections 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 --2- 172.21.15.105:0/464798164 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c06bb70 0x7f968c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 --2- 172.21.15.105:0/464798164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c108a50 0x7f968c108e30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 --2- 172.21.15.105:0/464798164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c06b1b0 0x7f968c06b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 -- 172.21.15.105:0/464798164 >> 172.21.15.105:0/464798164 conn(0x7f968c077990 msgr2=0x7f968c077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 -- 172.21.15.105:0/464798164 shutdown_connections 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.875+0000 7f9692fca700 1 -- 172.21.15.105:0/464798164 wait complete. 2024-04-22T06:36:15.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 Processor -- start 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 -- start start 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c06b1b0 0x7f968c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 0x7f968c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c108a50 0x7f968c19da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f968c112c90 con 0x7f968c06bb70 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f968c112b10 con 0x7f968c108a50 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9692fca700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f968c112e10 con 0x7f968c06b1b0 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 0x7f968c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 0x7f968c1996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42916/0 (socket says 172.21.15.105:42916) 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 -- 172.21.15.105:0/1903541076 learned_addr learned my addr 172.21.15.105:0/1903541076 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9690d66700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c06b1b0 0x7f968c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9691567700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c108a50 0x7f968c19da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 -- 172.21.15.105:0/1903541076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c06b1b0 msgr2=0x7f968c199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c06b1b0 0x7f968c199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 -- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c108a50 msgr2=0x7f968c19da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c108a50 0x7f968c19da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c19e140 con 0x7f968c06bb70 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9691567700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c108a50 0x7f968c19da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:15.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.876+0000 7f9683fff700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 0x7f968c1996c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f967c00d980 tx=0x7f967c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c016070 con 0x7f968c06bb70 2024-04-22T06:36:15.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968c19e430 con 0x7f968c06bb70 2024-04-22T06:36:15.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f967c004830 con 0x7f968c06bb70 2024-04-22T06:36:15.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c0054c0 con 0x7f968c06bb70 2024-04-22T06:36:15.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f968c1a5cb0 con 0x7f968c06bb70 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.877+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f968c0631d0 con 0x7f968c06bb70 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f967c004220 con 0x7f968c06bb70 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f96740a3cd0 0x7f96740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9690d66700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f96740a3cd0 0x7f96740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f967c0d0890 con 0x7f968c06bb70 2024-04-22T06:36:15.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] conn(0x7f96740ad4b0 0x7f96740af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f967c015ec0 con 0x7f96740ad4b0 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.879+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f967c0d0d70 con 0x7f968c06bb70 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.880+0000 7f9690d66700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f96740a3cd0 0x7f96740a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9688009e60 tx=0x7f968800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.880+0000 7f9691567700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] conn(0x7f96740ad4b0 0x7f96740af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.880+0000 7f9691567700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] conn(0x7f96740ad4b0 0x7f96740af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:15.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.881+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== osd.4 v2:172.21.15.112:6808/2758455817 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f967c015ec0 con 0x7f96740ad4b0 2024-04-22T06:36:15.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.897+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 --> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f968c063650 con 0x7f96740ad4b0 2024-04-22T06:36:15.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.897+0000 7f9681ffb700 1 -- 172.21.15.105:0/1903541076 <== osd.4 v2:172.21.15.112:6808/2758455817 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f968c063650 con 0x7f96740ad4b0 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] conn(0x7f96740ad4b0 msgr2=0x7f96740af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] conn(0x7f96740ad4b0 0x7f96740af8d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f96740a3cd0 msgr2=0x7f96740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f96740a3cd0 0x7f96740a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9688009e60 tx=0x7f968800b7b0 comp rx=0 tx=0).stop 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 msgr2=0x7f968c1996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 --2- 172.21.15.105:0/1903541076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f968c06bb70 0x7f968c1996c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f967c00d980 tx=0x7f967c00dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9691567700 1 -- 172.21.15.105:0/1903541076 reap_dead start 2024-04-22T06:36:15.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 shutdown_connections 2024-04-22T06:36:15.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 >> 172.21.15.105:0/1903541076 conn(0x7f968c077990 msgr2=0x7f968c0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:15.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 shutdown_connections 2024-04-22T06:36:15.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:15.898+0000 7f9692fca700 1 -- 172.21.15.105:0/1903541076 wait complete. 2024-04-22T06:36:15.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:15 smithi177 ceph-mon[31346]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 -- 172.21.15.105:0/740756221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190108ad0 msgr2=0x7fa190108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 --2- 172.21.15.105:0/740756221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190108ad0 0x7fa190108eb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa184009990 tx=0x7fa184009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 -- 172.21.15.105:0/740756221 shutdown_connections 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 --2- 172.21.15.105:0/740756221 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1901019d0 0x7fa19010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 --2- 172.21.15.105:0/740756221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190108ad0 0x7fa190108eb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 --2- 172.21.15.105:0/740756221 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa190101030 0x7fa190101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 -- 172.21.15.105:0/740756221 >> 172.21.15.105:0/740756221 conn(0x7fa190077fb0 msgr2=0x7fa1900ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 -- 172.21.15.105:0/740756221 shutdown_connections 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.059+0000 7fa196dd2700 1 -- 172.21.15.105:0/740756221 wait complete. 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 Processor -- start 2024-04-22T06:36:16.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 -- start start 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 0x7fa190199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1901019d0 0x7fa190199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa190108ad0 0x7fa19019da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112be0 con 0x7fa190101030 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112a60 con 0x7fa190108ad0 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa196dd2700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa190112d60 con 0x7fa1901019d0 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa194b6e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 0x7fa190199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa194b6e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 0x7fa190199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42944/0 (socket says 172.21.15.105:42944) 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa194b6e700 1 -- 172.21.15.105:0/3227771659 learned_addr learned my addr 172.21.15.105:0/3227771659 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa18ffff700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1901019d0 0x7fa190199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.060+0000 7fa19536f700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa190108ad0 0x7fa19019da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 -- 172.21.15.105:0/3227771659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1901019d0 msgr2=0x7fa190199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa1901019d0 0x7fa190199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 -- 172.21.15.105:0/3227771659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa190108ad0 msgr2=0x7fa19019da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa190108ad0 0x7fa19019da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19019e140 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa194b6e700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 0x7fa190199130 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184027070 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19019e3d0 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa184022470 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184017650 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.061+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1901a5cb0 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.062+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa1900631d0 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.063+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa18402a070 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18dffb700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1800a3cd0 0x7fa1800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18ffff700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1800a3cd0 0x7fa1800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fa1840d19f0 con 0x7fa190101030 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18dffb700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] conn(0x7fa1800ad4b0 0x7fa1800af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1800aff80 con 0x7fa1800ad4b0 2024-04-22T06:36:16.068 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa1840d1ed0 con 0x7fa190101030 2024-04-22T06:36:16.069 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa18ffff700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1800a3cd0 0x7fa1800a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa1901082a0 tx=0x7fa17c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.069 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa19536f700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] conn(0x7fa1800ad4b0 0x7fa1800af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.069 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.064+0000 7fa19536f700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] conn(0x7fa1800ad4b0 0x7fa1800af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.069 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.065+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== osd.9 v2:172.21.15.105:6826/716531130 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1800aff80 con 0x7fa1800ad4b0 2024-04-22T06:36:16.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 --> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa1900635d0 con 0x7fa1800ad4b0 2024-04-22T06:36:16.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa18dffb700 1 -- 172.21.15.105:0/3227771659 <== osd.9 v2:172.21.15.105:6826/716531130 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa1900635d0 con 0x7fa1800ad4b0 2024-04-22T06:36:16.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] conn(0x7fa1800ad4b0 msgr2=0x7fa1800af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] conn(0x7fa1800ad4b0 0x7fa1800af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1800a3cd0 msgr2=0x7fa1800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa1800a3cd0 0x7fa1800a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa1901082a0 tx=0x7fa17c009260 comp rx=0 tx=0).stop 2024-04-22T06:36:16.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 msgr2=0x7fa190199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 --2- 172.21.15.105:0/3227771659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa190101030 0x7fa190199130 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184009580 comp rx=0 tx=0).stop 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa19536f700 1 -- 172.21.15.105:0/3227771659 reap_dead start 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.082+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 shutdown_connections 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.083+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 >> 172.21.15.105:0/3227771659 conn(0x7fa190077fb0 msgr2=0x7fa1900ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.083+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 shutdown_connections 2024-04-22T06:36:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.083+0000 7fa196dd2700 1 -- 172.21.15.105:0/3227771659 wait complete. 2024-04-22T06:36:16.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:15 smithi112 ceph-mon[31236]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 -- 172.21.15.105:0/1874256255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560102ac0 msgr2=0x7fa560102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 --2- 172.21.15.105:0/1874256255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560102ac0 0x7fa560102f40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa554009990 tx=0x7fa554009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 -- 172.21.15.105:0/1874256255 shutdown_connections 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 --2- 172.21.15.105:0/1874256255 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa560108a70 0x7fa560108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 --2- 172.21.15.105:0/1874256255 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa560103480 0x7fa56010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 --2- 172.21.15.105:0/1874256255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560102ac0 0x7fa560102f40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 -- 172.21.15.105:0/1874256255 >> 172.21.15.105:0/1874256255 conn(0x7fa560079bb0 msgr2=0x7fa560079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 -- 172.21.15.105:0/1874256255 shutdown_connections 2024-04-22T06:36:16.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.708+0000 7fa565142700 1 -- 172.21.15.105:0/1874256255 wait complete. 2024-04-22T06:36:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 Processor -- start 2024-04-22T06:36:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 -- start start 2024-04-22T06:36:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa560102ac0 0x7fa5601991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa560103480 0x7fa560199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 0x7fa56019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa560112c80 con 0x7fa560108a70 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa560112b00 con 0x7fa560103480 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa565142700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa560112e00 con 0x7fa560102ac0 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa55f59e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 0x7fa56019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa55f59e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 0x7fa56019daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42954/0 (socket says 172.21.15.105:42954) 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.709+0000 7fa55f59e700 1 -- 172.21.15.105:0/1459700551 learned_addr learned my addr 172.21.15.105:0/1459700551 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55e59c700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa560103480 0x7fa560199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55ed9d700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa560102ac0 0x7fa5601991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 -- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa560102ac0 msgr2=0x7fa5601991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa560102ac0 0x7fa5601991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 -- 172.21.15.105:0/1459700551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa560103480 msgr2=0x7fa560199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa560103480 0x7fa560199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56019e1e0 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa55f59e700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 0x7fa56019daa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa558007ee0 tx=0x7fa55800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55800dbd0 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa55800dd70 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558005990 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56019e4d0 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.710+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5601a5d50 con 0x7fa560108a70 2024-04-22T06:36:16.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.711+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa560063150 con 0x7fa560108a70 2024-04-22T06:36:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.712+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa55801a020 con 0x7fa560108a70 2024-04-22T06:36:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa547fff700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5480a3d50 0x7fa5480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa55ed9d700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5480a3d50 0x7fa5480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fa5580d01f0 con 0x7fa560108a70 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa547fff700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] conn(0x7fa5480ad530 0x7fa5480af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5480b0000 con 0x7fa5480ad530 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa5580d06d0 con 0x7fa560108a70 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa55ed9d700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5480a3d50 0x7fa5480a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa550006fd0 tx=0x7fa550008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.713+0000 7fa55e59c700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] conn(0x7fa5480ad530 0x7fa5480af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.714+0000 7fa55e59c700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] conn(0x7fa5480ad530 0x7fa5480af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.715+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== osd.11 v2:172.21.15.177:6824/1875698423 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5480b0000 con 0x7fa5480ad530 2024-04-22T06:36:16.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.731+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 --> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa5600635d0 con 0x7fa5480ad530 2024-04-22T06:36:16.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa547fff700 1 -- 172.21.15.105:0/1459700551 <== osd.11 v2:172.21.15.177:6824/1875698423 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa5600635d0 con 0x7fa5480ad530 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] conn(0x7fa5480ad530 msgr2=0x7fa5480af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] conn(0x7fa5480ad530 0x7fa5480af950 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5480a3d50 msgr2=0x7fa5480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5480a3d50 0x7fa5480a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa550006fd0 tx=0x7fa550008040 comp rx=0 tx=0).stop 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 msgr2=0x7fa56019daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 --2- 172.21.15.105:0/1459700551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa560108a70 0x7fa56019daa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa558007ee0 tx=0x7fa55800d3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:16.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa55f59e700 1 -- 172.21.15.105:0/1459700551 reap_dead start 2024-04-22T06:36:16.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 shutdown_connections 2024-04-22T06:36:16.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 >> 172.21.15.105:0/1459700551 conn(0x7fa560079bb0 msgr2=0x7fa5601018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:16.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 shutdown_connections 2024-04-22T06:36:16.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:16.732+0000 7fa565142700 1 -- 172.21.15.105:0/1459700551 wait complete. 2024-04-22T06:36:17.168 INFO:teuthology.orchestra.run.smithi105.stdout:326417514520 2024-04-22T06:36:17.168 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-22T06:36:17.845 INFO:teuthology.orchestra.run.smithi105.stdout:146028888119 2024-04-22T06:36:17.846 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T06:36:17.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:17 smithi177 ceph-mon[31346]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:18.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:17 smithi105 ceph-mon[27916]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:18.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:17 smithi112 ceph-mon[31236]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:18.557 INFO:teuthology.orchestra.run.smithi105.stdout:300647710751 2024-04-22T06:36:18.557 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-22T06:36:19.549 INFO:teuthology.orchestra.run.smithi105.stdout:360777252885 2024-04-22T06:36:19.550 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-22T06:36:19.909 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:19 smithi105 ceph-mon[27916]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:19.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:19 smithi177 ceph-mon[31346]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 -- 172.21.15.105:0/2954236494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff520107940 msgr2=0x7ff520107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 --2- 172.21.15.105:0/2954236494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff520107940 0x7ff520107d20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff514009990 tx=0x7ff514009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 -- 172.21.15.105:0/2954236494 shutdown_connections 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 --2- 172.21.15.105:0/2954236494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 0x7ff52010e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 --2- 172.21.15.105:0/2954236494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff520107940 0x7ff520107d20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 --2- 172.21.15.105:0/2954236494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5201019d0 0x7ff520101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 -- 172.21.15.105:0/2954236494 >> 172.21.15.105:0/2954236494 conn(0x7ff520077990 msgr2=0x7ff520077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 -- 172.21.15.105:0/2954236494 shutdown_connections 2024-04-22T06:36:20.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.116+0000 7ff526e7d700 1 -- 172.21.15.105:0/2954236494 wait complete. 2024-04-22T06:36:20.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 Processor -- start 2024-04-22T06:36:20.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 -- start start 2024-04-22T06:36:20.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 0x7ff520103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 0x7ff520103a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff520107940 0x7ff520103f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111b80 con 0x7ff5201019d0 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111a00 con 0x7ff520107940 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff526e7d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111d00 con 0x7ff520102390 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff524c19700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 0x7ff520103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff524c19700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 0x7ff520103510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42968/0 (socket says 172.21.15.105:42968) 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff524c19700 1 -- 172.21.15.105:0/1861383115 learned_addr learned my addr 172.21.15.105:0/1861383115 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff52541a700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff520107940 0x7ff520103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff51ffff700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 0x7ff520103a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.117+0000 7ff524c19700 1 -- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 msgr2=0x7ff520103a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff524c19700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 0x7ff520103a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff524c19700 1 -- 172.21.15.105:0/1861383115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff520107940 msgr2=0x7ff520103f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff524c19700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff520107940 0x7ff520103f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff524c19700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5201a76e0 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff51ffff700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff520102390 0x7ff520103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff524c19700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 0x7ff520103510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff514000c00 tx=0x7ff5140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514027070 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff514003fa0 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5201a7970 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514022db0 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.118+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201a7dc0 con 0x7ff5201019d0 2024-04-22T06:36:20.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.119+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff5200630d0 con 0x7ff5201019d0 2024-04-22T06:36:20.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.120+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff51402a070 con 0x7ff5201019d0 2024-04-22T06:36:20.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51dffb700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5100a3cd0 0x7ff5100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51ffff700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5100a3cd0 0x7ff5100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7ff5140d1d70 con 0x7ff5201019d0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51dffb700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] conn(0x7ff5100ad4b0 0x7ff5100af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff514017e50 con 0x7ff5100ad4b0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff5140d2120 con 0x7ff5201019d0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.121+0000 7ff51ffff700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5100a3cd0 0x7ff5100a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5200780d0 tx=0x7ff50c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.122+0000 7ff52541a700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] conn(0x7ff5100ad4b0 0x7ff5100af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.122+0000 7ff52541a700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] conn(0x7ff5100ad4b0 0x7ff5100af8d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.123+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== osd.5 v2:172.21.15.177:6808/1059606391 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff514017e50 con 0x7ff5100ad4b0 2024-04-22T06:36:20.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.139+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 --> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff5200634d0 con 0x7ff5100ad4b0 2024-04-22T06:36:20.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff51dffb700 1 -- 172.21.15.105:0/1861383115 <== osd.5 v2:172.21.15.177:6808/1059606391 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff5200634d0 con 0x7ff5100ad4b0 2024-04-22T06:36:20.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] conn(0x7ff5100ad4b0 msgr2=0x7ff5100af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] conn(0x7ff5100ad4b0 0x7ff5100af8d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5100a3cd0 msgr2=0x7ff5100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5100a3cd0 0x7ff5100a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff5200780d0 tx=0x7ff50c008040 comp rx=0 tx=0).stop 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 msgr2=0x7ff520103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 --2- 172.21.15.105:0/1861383115 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5201019d0 0x7ff520103510 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff514000c00 tx=0x7ff5140047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff52541a700 1 -- 172.21.15.105:0/1861383115 reap_dead start 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 shutdown_connections 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 >> 172.21.15.105:0/1861383115 conn(0x7ff520077990 msgr2=0x7ff52006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 shutdown_connections 2024-04-22T06:36:20.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.140+0000 7ff526e7d700 1 -- 172.21.15.105:0/1861383115 wait complete. 2024-04-22T06:36:20.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:19 smithi112 ceph-mon[31236]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.981+0000 7f647817a700 1 -- 172.21.15.105:0/2688930913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f647006b980 msgr2=0x7f647010f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.981+0000 7f647817a700 1 --2- 172.21.15.105:0/2688930913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f647006b980 0x7f647010f8f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f646800b0a0 tx=0x7f646800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.981+0000 7f647817a700 1 -- 172.21.15.105:0/2688930913 shutdown_connections 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 --2- 172.21.15.105:0/2688930913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f647006b980 0x7f647010f8f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 --2- 172.21.15.105:0/2688930913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6470108a20 0x7f6470108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 --2- 172.21.15.105:0/2688930913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 0x7f647006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 -- 172.21.15.105:0/2688930913 >> 172.21.15.105:0/2688930913 conn(0x7f64700788b0 msgr2=0x7f6470078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 -- 172.21.15.105:0/2688930913 shutdown_connections 2024-04-22T06:36:20.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 -- 172.21.15.105:0/2688930913 wait complete. 2024-04-22T06:36:20.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 Processor -- start 2024-04-22T06:36:20.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.982+0000 7f647817a700 1 -- start start 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 0x7f6470101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f647006b980 0x7f6470101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 0x7f6470102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6470112c70 con 0x7f6470108a20 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6470112af0 con 0x7f647006afe0 2024-04-22T06:36:20.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f647817a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6470112df0 con 0x7f647006b980 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 0x7f6470102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 0x7f6470102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42984/0 (socket says 172.21.15.105:42984) 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 -- 172.21.15.105:0/2886041780 learned_addr learned my addr 172.21.15.105:0/2886041780 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6475715700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f647006b980 0x7f6470101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6475f16700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 0x7f6470101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 -- 172.21.15.105:0/2886041780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f647006b980 msgr2=0x7f6470101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f647006b980 0x7f6470101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 -- 172.21.15.105:0/2886041780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 msgr2=0x7f6470101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 0x7f6470101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701a7610 con 0x7f6470108a20 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6475f16700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f647006afe0 0x7f6470101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.983+0000 7f6476717700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 0x7f6470102480 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f646800b070 tx=0x7f646800cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.984+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468021070 con 0x7f6470108a20 2024-04-22T06:36:20.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.984+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6468007cf0 con 0x7f6470108a20 2024-04-22T06:36:20.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.984+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64701a78a0 con 0x7f6470108a20 2024-04-22T06:36:20.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.984+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701a7d50 con 0x7f6470108a20 2024-04-22T06:36:20.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.984+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64680053c0 con 0x7f6470108a20 2024-04-22T06:36:20.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.985+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6470063110 con 0x7f6470108a20 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.986+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6468024070 con 0x7f6470108a20 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6466ffd700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f645c0a3e20 0x7f645c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6475f16700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f645c0a3e20 0x7f645c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f646800f070 con 0x7f6470108a20 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6466ffd700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] conn(0x7f645c0ad660 0x7f645c0afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64680020b0 con 0x7f645c0ad660 2024-04-22T06:36:20.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f64680d1d70 con 0x7f6470108a20 2024-04-22T06:36:20.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6475715700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] conn(0x7f645c0ad660 0x7f645c0afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:20.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6475f16700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f645c0a3e20 0x7f645c0a62e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f64600099d0 tx=0x7f6460008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.987+0000 7f6475715700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] conn(0x7f645c0ad660 0x7f645c0afa80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:20.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:20.988+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== osd.6 v2:172.21.15.105:6818/546897953 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f64680020b0 con 0x7f645c0ad660 2024-04-22T06:36:21.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.005+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 --> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6470063510 con 0x7f645c0ad660 2024-04-22T06:36:21.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.005+0000 7f6466ffd700 1 -- 172.21.15.105:0/2886041780 <== osd.6 v2:172.21.15.105:6818/546897953 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6470063510 con 0x7f645c0ad660 2024-04-22T06:36:21.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] conn(0x7f645c0ad660 msgr2=0x7f645c0afa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] conn(0x7f645c0ad660 0x7f645c0afa80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:21.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f645c0a3e20 msgr2=0x7f645c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f645c0a3e20 0x7f645c0a62e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f64600099d0 tx=0x7f6460008040 comp rx=0 tx=0).stop 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 msgr2=0x7f6470102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 --2- 172.21.15.105:0/2886041780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6470108a20 0x7f6470102480 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f646800b070 tx=0x7f646800cf00 comp rx=0 tx=0).stop 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f6476717700 1 -- 172.21.15.105:0/2886041780 reap_dead start 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 shutdown_connections 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 >> 172.21.15.105:0/2886041780 conn(0x7f64700788b0 msgr2=0x7f64700ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 shutdown_connections 2024-04-22T06:36:21.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:21.006+0000 7f647817a700 1 -- 172.21.15.105:0/2886041780 wait complete. 2024-04-22T06:36:21.747 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:21 smithi105 ceph-mon[27916]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:21.790 INFO:teuthology.orchestra.run.smithi105.stdout:34359738443 2024-04-22T06:36:21.791 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T06:36:21.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:21 smithi177 ceph-mon[31346]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:21 smithi112 ceph-mon[31236]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:22.229 INFO:teuthology.orchestra.run.smithi105.stdout:206158430255 2024-04-22T06:36:22.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 -- 172.21.15.105:0/1049616937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 msgr2=0x7eff1006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 --2- 172.21.15.105:0/1049616937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff1006b5f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7eff00009990 tx=0x7eff00009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 -- 172.21.15.105:0/1049616937 shutdown_connections 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 --2- 172.21.15.105:0/1049616937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff10108a30 0x7eff10108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 --2- 172.21.15.105:0/1049616937 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff1006bb30 0x7eff1010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 --2- 172.21.15.105:0/1049616937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff1006b5f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 -- 172.21.15.105:0/1049616937 >> 172.21.15.105:0/1049616937 conn(0x7eff10078910 msgr2=0x7eff10078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 -- 172.21.15.105:0/1049616937 shutdown_connections 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.133+0000 7eff174e9700 1 -- 172.21.15.105:0/1049616937 wait complete. 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 Processor -- start 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 -- start start 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff10199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff1006bb30 0x7eff10199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff10108a30 0x7eff1019da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff10112c10 con 0x7eff1006b190 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7eff10112a90 con 0x7eff1006bb30 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff174e9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7eff10112d90 con 0x7eff10108a30 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff15285700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff10199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff15285700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff10199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43012/0 (socket says 172.21.15.105:43012) 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.134+0000 7eff15285700 1 -- 172.21.15.105:0/1879057437 learned_addr learned my addr 172.21.15.105:0/1879057437 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:23.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15a86700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff10108a30 0x7eff1019da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff14a84700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff1006bb30 0x7eff10199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 -- 172.21.15.105:0/1879057437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff10108a30 msgr2=0x7eff1019da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff10108a30 0x7eff1019da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 -- 172.21.15.105:0/1879057437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff1006bb30 msgr2=0x7eff10199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff1006bb30 0x7eff10199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1019e150 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff15285700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff10199140 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00c9b0 tx=0x7eff0c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c007b00 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1019e440 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff101a5cc0 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff0c004ce0 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.135+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c005530 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.136+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efef4001030 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.137+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7eff0c004750 con 0x7eff1006b190 2024-04-22T06:36:23.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff067fc700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efefc0a3d50 0x7efefc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff14a84700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efefc0a3d50 0x7efefc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7eff0c0d0630 con 0x7eff1006b190 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff067fc700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] conn(0x7efefc0ad530 0x7efefc0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff0c011ec0 con 0x7efefc0ad530 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7eff0c0d0b10 con 0x7eff1006b190 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff15a86700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] conn(0x7efefc0ad530 0x7efefc0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff14a84700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efefc0a3d50 0x7efefc0a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7eff00006010 tx=0x7eff0001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.138+0000 7eff15a86700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] conn(0x7efefc0ad530 0x7efefc0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.139+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== osd.3 v2:172.21.15.105:6810/3727777768 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff0c011ec0 con 0x7efefc0ad530 2024-04-22T06:36:23.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.156+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 --> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efef4002e60 con 0x7efefc0ad530 2024-04-22T06:36:23.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff067fc700 1 -- 172.21.15.105:0/1879057437 <== osd.3 v2:172.21.15.105:6810/3727777768 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efef4002e60 con 0x7efefc0ad530 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] conn(0x7efefc0ad530 msgr2=0x7efefc0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] conn(0x7efefc0ad530 0x7efefc0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efefc0a3d50 msgr2=0x7efefc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7efefc0a3d50 0x7efefc0a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7eff00006010 tx=0x7eff0001a040 comp rx=0 tx=0).stop 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 msgr2=0x7eff10199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 --2- 172.21.15.105:0/1879057437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff1006b190 0x7eff10199140 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00c9b0 tx=0x7eff0c00ccc0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff15a86700 1 -- 172.21.15.105:0/1879057437 reap_dead start 2024-04-22T06:36:23.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 shutdown_connections 2024-04-22T06:36:23.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 >> 172.21.15.105:0/1879057437 conn(0x7eff10078910 msgr2=0x7eff100ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 shutdown_connections 2024-04-22T06:36:23.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.157+0000 7eff174e9700 1 -- 172.21.15.105:0/1879057437 wait complete. 2024-04-22T06:36:23.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 -- 172.21.15.105:0/2628637262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06afe0 msgr2=0x7ff46c06b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 --2- 172.21.15.105:0/2628637262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06afe0 0x7ff46c06b440 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff460009990 tx=0x7ff460009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 -- 172.21.15.105:0/2628637262 shutdown_connections 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 --2- 172.21.15.105:0/2628637262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06b980 0x7ff46c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 --2- 172.21.15.105:0/2628637262 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46c108a20 0x7ff46c108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 --2- 172.21.15.105:0/2628637262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06afe0 0x7ff46c06b440 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 -- 172.21.15.105:0/2628637262 >> 172.21.15.105:0/2628637262 conn(0x7ff46c0788b0 msgr2=0x7ff46c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 -- 172.21.15.105:0/2628637262 shutdown_connections 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.360+0000 7ff472512700 1 -- 172.21.15.105:0/2628637262 wait complete. 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 Processor -- start 2024-04-22T06:36:23.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 -- start start 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06afe0 0x7ff46c199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 0x7ff46c1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46c108a20 0x7ff46c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c112c20 con 0x7ff46c06b980 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c112aa0 con 0x7ff46c06afe0 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff472512700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c112da0 con 0x7ff46c108a20 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff46b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 0x7ff46c1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.361+0000 7ff46b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 0x7ff46c1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43024/0 (socket says 172.21.15.105:43024) 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 -- 172.21.15.105:0/3717995892 learned_addr learned my addr 172.21.15.105:0/3717995892 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46bfff700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06afe0 0x7ff46c199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff470aaf700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46c108a20 0x7ff46c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 -- 172.21.15.105:0/3717995892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46c108a20 msgr2=0x7ff46c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff46c108a20 0x7ff46c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 -- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06afe0 msgr2=0x7ff46c199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06afe0 0x7ff46c199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c19e1a0 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46bfff700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff46c06afe0 0x7ff46c199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff46b7fe700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 0x7ff46c1996d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff460000c00 tx=0x7ff4600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460027070 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c19e430 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c1a5d10 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4600041d0 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.362+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46000f460 con 0x7ff46c06b980 2024-04-22T06:36:23.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.363+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff46c0631d0 con 0x7ff46c06b980 2024-04-22T06:36:23.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.364+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff46002a070 con 0x7ff46c06b980 2024-04-22T06:36:23.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff4697fa700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff4540a3cd0 0x7ff4540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff46bfff700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff4540a3cd0 0x7ff4540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7ff4600d1a90 con 0x7ff46c06b980 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff4697fa700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] conn(0x7ff4540ad4b0 0x7ff4540af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff46002de50 con 0x7ff4540ad4b0 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff4600d1f70 con 0x7ff46c06b980 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff46bfff700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff4540a3cd0 0x7ff4540a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0099d0 tx=0x7ff45c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.365+0000 7ff470aaf700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] conn(0x7ff4540ad4b0 0x7ff4540af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.366+0000 7ff470aaf700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] conn(0x7ff4540ad4b0 0x7ff4540af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.366+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== osd.1 v2:172.21.15.112:6800/3644741495 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff46002de50 con 0x7ff4540ad4b0 2024-04-22T06:36:23.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.383+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 --> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff46c0635d0 con 0x7ff4540ad4b0 2024-04-22T06:36:23.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.383+0000 7ff4697fa700 1 -- 172.21.15.105:0/3717995892 <== osd.1 v2:172.21.15.112:6800/3644741495 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff46c0635d0 con 0x7ff4540ad4b0 2024-04-22T06:36:23.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] conn(0x7ff4540ad4b0 msgr2=0x7ff4540af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] conn(0x7ff4540ad4b0 0x7ff4540af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff4540a3cd0 msgr2=0x7ff4540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff4540a3cd0 0x7ff4540a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0099d0 tx=0x7ff45c008040 comp rx=0 tx=0).stop 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 msgr2=0x7ff46c1996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 --2- 172.21.15.105:0/3717995892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff46c06b980 0x7ff46c1996d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff460000c00 tx=0x7ff4600047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff470aaf700 1 -- 172.21.15.105:0/3717995892 reap_dead start 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 shutdown_connections 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 >> 172.21.15.105:0/3717995892 conn(0x7ff46c0788b0 msgr2=0x7ff46c0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 shutdown_connections 2024-04-22T06:36:23.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.384+0000 7ff472512700 1 -- 172.21.15.105:0/3717995892 wait complete. 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 -- 172.21.15.105:0/1005374173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 msgr2=0x7fcacc102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 --2- 172.21.15.105:0/1005374173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc102f10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009990 tx=0x7fcabc009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 -- 172.21.15.105:0/1005374173 shutdown_connections 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 --2- 172.21.15.105:0/1005374173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcacc103450 0x7fcacc10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 --2- 172.21.15.105:0/1005374173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc108a80 0x7fcacc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 --2- 172.21.15.105:0/1005374173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc102f10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.670+0000 7fcad328b700 1 -- 172.21.15.105:0/1005374173 >> 172.21.15.105:0/1005374173 conn(0x7fcacc079bf0 msgr2=0x7fcacc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- 172.21.15.105:0/1005374173 shutdown_connections 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- 172.21.15.105:0/1005374173 wait complete. 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 Processor -- start 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- start start 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc103450 0x7fcacc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcacc108a80 0x7fcacc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc112c30 con 0x7fcacc102a90 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc112ab0 con 0x7fcacc108a80 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad328b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcacc112db0 con 0x7fcacc103450 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.671+0000 7fcad1027700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43032/0 (socket says 172.21.15.105:43032) 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 -- 172.21.15.105:0/725324561 learned_addr learned my addr 172.21.15.105:0/725324561 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad0826700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc103450 0x7fcacc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1828700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcacc108a80 0x7fcacc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 -- 172.21.15.105:0/725324561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc103450 msgr2=0x7fcacc075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcacc103450 0x7fcacc075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 -- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcacc108a80 msgr2=0x7fcacc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcacc108a80 0x7fcacc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcacc1a7630 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad1027700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc074d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d980 tx=0x7fcac800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8017070 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcac8005020 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad328b700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcacc1a7920 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac800fa10 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.672+0000 7fcad328b700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcacc1a7e30 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.673+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcacc0630d0 con 0x7fcacc102a90 2024-04-22T06:36:23.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.674+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fcac8004a30 con 0x7fcacc102a90 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.675+0000 7fcac27fc700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcab80a3d50 0x7fcab80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.675+0000 7fcad0826700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcab80a3d50 0x7fcab80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.675+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fcac80d08a0 con 0x7fcacc102a90 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcac27fc700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] conn(0x7fcab80ad530 0x7fcab80af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcab80b0000 con 0x7fcab80ad530 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fcac80d0c50 con 0x7fcacc102a90 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcad0826700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcab80a3d50 0x7fcab80a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00b5c0 tx=0x7fcabc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcad1828700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] conn(0x7fcab80ad530 0x7fcab80af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.676+0000 7fcad1828700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] conn(0x7fcab80ad530 0x7fcab80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:23.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.677+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== osd.7 v2:172.21.15.112:6816/1044643944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcab80b0000 con 0x7fcab80ad530 2024-04-22T06:36:23.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.693+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 --> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcacc063550 con 0x7fcab80ad530 2024-04-22T06:36:23.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.693+0000 7fcac27fc700 1 -- 172.21.15.105:0/725324561 <== osd.7 v2:172.21.15.112:6816/1044643944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcacc063550 con 0x7fcab80ad530 2024-04-22T06:36:23.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] conn(0x7fcab80ad530 msgr2=0x7fcab80af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] conn(0x7fcab80ad530 0x7fcab80af950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcab80a3d50 msgr2=0x7fcab80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcab80a3d50 0x7fcab80a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00b5c0 tx=0x7fcabc005960 comp rx=0 tx=0).stop 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 msgr2=0x7fcacc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 --2- 172.21.15.105:0/725324561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcacc102a90 0x7fcacc074d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d980 tx=0x7fcac800dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcad1828700 1 -- 172.21.15.105:0/725324561 reap_dead start 2024-04-22T06:36:23.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 shutdown_connections 2024-04-22T06:36:23.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 >> 172.21.15.105:0/725324561 conn(0x7fcacc079bf0 msgr2=0x7fcacc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:23.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 shutdown_connections 2024-04-22T06:36:23.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:23.694+0000 7fcab7fff700 1 -- 172.21.15.105:0/725324561 wait complete. 2024-04-22T06:36:23.876 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:23.876 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:23.877 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:23.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:23 smithi105 ceph-mon[27916]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-22T06:36:23.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:23 smithi177 ceph-mon[31346]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-22T06:36:24.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:23 smithi112 ceph-mon[31236]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-22T06:36:24.641 INFO:teuthology.orchestra.run.smithi105.stdout:171798691890 2024-04-22T06:36:24.642 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T06:36:24.825 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:25.246 INFO:teuthology.orchestra.run.smithi105.stdout:55834574919 2024-04-22T06:36:25.246 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T06:36:25.868 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:25 smithi105 ceph-mon[27916]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T06:36:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:25 smithi177 ceph-mon[31346]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T06:36:26.011 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:25 smithi112 ceph-mon[31236]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-04-22T06:36:26.174 INFO:teuthology.orchestra.run.smithi105.stdout:111669149755 2024-04-22T06:36:26.175 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T06:36:26.695 INFO:teuthology.orchestra.run.smithi105.stdout:227633266730 2024-04-22T06:36:26.695 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T06:36:27.321 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:27.321 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:27.321 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:27.874 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:27 smithi105 ceph-mon[27916]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:27 smithi177 ceph-mon[31346]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:28.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:27 smithi112 ceph-mon[31236]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:29.871 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:29 smithi105 ceph-mon[27916]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T06:36:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:29 smithi177 ceph-mon[31346]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.166+0000 7f681c63b700 1 -- 172.21.15.105:0/1019403585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814107980 msgr2=0x7f6814107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.166+0000 7f681c63b700 1 --2- 172.21.15.105:0/1019403585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814107980 0x7f6814107d60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6810009990 tx=0x7f6810009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 -- 172.21.15.105:0/1019403585 shutdown_connections 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 --2- 172.21.15.105:0/1019403585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f68141022f0 0x7f681410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 --2- 172.21.15.105:0/1019403585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814107980 0x7f6814107d60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 --2- 172.21.15.105:0/1019403585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6814101930 0x7f6814101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 -- 172.21.15.105:0/1019403585 >> 172.21.15.105:0/1019403585 conn(0x7f68140788b0 msgr2=0x7f6814078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:30.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 -- 172.21.15.105:0/1019403585 shutdown_connections 2024-04-22T06:36:30.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 -- 172.21.15.105:0/1019403585 wait complete. 2024-04-22T06:36:30.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 Processor -- start 2024-04-22T06:36:30.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.167+0000 7f681c63b700 1 -- start start 2024-04-22T06:36:30.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68141022f0 0x7f6814199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6814107980 0x7f681419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6814111ae0 con 0x7f6814101930 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6814111960 con 0x7f68141022f0 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681c63b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6814111c60 con 0x7f6814107980 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34970/0 (socket says 172.21.15.105:34970) 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 -- 172.21.15.105:0/1673934085 learned_addr learned my addr 172.21.15.105:0/1673934085 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f6819bd6700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68141022f0 0x7f6814199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681abd8700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6814107980 0x7f681419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 -- 172.21.15.105:0/1673934085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6814107980 msgr2=0x7f681419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6814107980 0x7f681419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 -- 172.21.15.105:0/1673934085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68141022f0 msgr2=0x7f6814199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68141022f0 0x7f6814199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 -- 172.21.15.105:0/1673934085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681419e100 con 0x7f6814101930 2024-04-22T06:36:30.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f681a3d7700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6810000c00 tx=0x7f68100047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.168+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810027070 con 0x7f6814101930 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.169+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681419e390 con 0x7f6814101930 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.169+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6810003f20 con 0x7f6814101930 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.169+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6810022ca0 con 0x7f6814101930 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.169+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68141a5c70 con 0x7f6814101930 2024-04-22T06:36:30.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.169+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6814069730 con 0x7f6814101930 2024-04-22T06:36:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.171+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f681002a070 con 0x7f6814101930 2024-04-22T06:36:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.171+0000 7f680b7fe700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 0x7f68000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.171+0000 7f6819bd6700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 0x7f68000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.171+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f68100d15f0 con 0x7f6814101930 2024-04-22T06:36:30.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.172+0000 7f6819bd6700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 0x7f68000a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6814078f50 tx=0x7f680400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:30.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.175+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6810031070 con 0x7f6814101930 2024-04-22T06:36:30.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:29 smithi112 ceph-mon[31236]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-22T06:36:30.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.385+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f68141a5e40 con 0x7f6814101930 2024-04-22T06:36:30.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.385+0000 7f680b7fe700 1 -- 172.21.15.105:0/1673934085 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f6810013070 con 0x7f6814101930 2024-04-22T06:36:30.384 INFO:teuthology.orchestra.run.smithi105.stdout:360777252887 2024-04-22T06:36:30.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.387+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 msgr2=0x7f68000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.387+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 0x7f68000a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6814078f50 tx=0x7f680400b410 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.387+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 msgr2=0x7f6814199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.387+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6810000c00 tx=0x7f68100047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 shutdown_connections 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f68000a3cd0 0x7f68000a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6814107980 0x7f681419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6814101930 0x7f6814199140 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 --2- 172.21.15.105:0/1673934085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f68141022f0 0x7f6814199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 >> 172.21.15.105:0/1673934085 conn(0x7f68140788b0 msgr2=0x7f681406b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 shutdown_connections 2024-04-22T06:36:30.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:30.388+0000 7f681c63b700 1 -- 172.21.15.105:0/1673934085 wait complete. 2024-04-22T06:36:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1673934085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:36:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:30 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1673934085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:36:31.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:30 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1673934085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-22T06:36:31.806 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252885 got 360777252887 for osd.11 2024-04-22T06:36:31.807 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:31.988 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:31 smithi105 ceph-mon[27916]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:31 smithi177 ceph-mon[31346]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:32.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:31 smithi112 ceph-mon[31236]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:32.530 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:33.990 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:33 smithi105 ceph-mon[27916]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:34.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:33 smithi112 ceph-mon[31236]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:34.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:33 smithi177 ceph-mon[31346]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.376+0000 7fc637c70700 1 -- 172.21.15.105:0/2740211345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc630108a50 msgr2=0x7fc630108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.376+0000 7fc637c70700 1 --2- 172.21.15.105:0/2740211345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc630108a50 0x7fc630108e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc628009990 tx=0x7fc628009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- 172.21.15.105:0/2740211345 shutdown_connections 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- 172.21.15.105:0/2740211345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006bb70 0x7fc63010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- 172.21.15.105:0/2740211345 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc630108a50 0x7fc630108e30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- 172.21.15.105:0/2740211345 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc63006b1b0 0x7fc63006b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- 172.21.15.105:0/2740211345 >> 172.21.15.105:0/2740211345 conn(0x7fc630077990 msgr2=0x7fc630077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- 172.21.15.105:0/2740211345 shutdown_connections 2024-04-22T06:36:35.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- 172.21.15.105:0/2740211345 wait complete. 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 Processor -- start 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- start start 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006b1b0 0x7fc63019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc630108a50 0x7fc63019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc630112c50 con 0x7fc63006bb70 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc630112ad0 con 0x7fc63006b1b0 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.377+0000 7fc637c70700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc630112dd0 con 0x7fc630108a50 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48082/0 (socket says 172.21.15.105:48082) 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 -- 172.21.15.105:0/4148880642 learned_addr learned my addr 172.21.15.105:0/4148880642 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc635a0c700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006b1b0 0x7fc63019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63620d700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc630108a50 0x7fc63019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 -- 172.21.15.105:0/4148880642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc630108a50 msgr2=0x7fc63019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc630108a50 0x7fc63019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 -- 172.21.15.105:0/4148880642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006b1b0 msgr2=0x7fc63019b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006b1b0 0x7fc63019b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 -- 172.21.15.105:0/4148880642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6301a03a0 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc63520b700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc62000d980 tx=0x7fc62000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620017070 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc620005020 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6301a0690 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62000fa10 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.378+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6301a7f10 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.379+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc630069730 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.380+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc620004a30 con 0x7fc63006bb70 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.381+0000 7fc626ffd700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 0x7fc61c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.381+0000 7fc635a0c700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 0x7fc61c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.381+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fc6200d0280 con 0x7fc63006bb70 2024-04-22T06:36:35.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.381+0000 7fc635a0c700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 0x7fc61c0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc62800b5c0 tx=0x7fc628005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:35.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.385+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc620020020 con 0x7fc63006bb70 2024-04-22T06:36:35.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.589+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc63002df80 con 0x7fc63006bb70 2024-04-22T06:36:35.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.589+0000 7fc626ffd700 1 -- 172.21.15.105:0/4148880642 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc62009ccb0 con 0x7fc63006bb70 2024-04-22T06:36:35.588 INFO:teuthology.orchestra.run.smithi105.stdout:206158430257 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 msgr2=0x7fc61c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 0x7fc61c0a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc62800b5c0 tx=0x7fc628005960 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 msgr2=0x7fc63019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc62000d980 tx=0x7fc62000dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 shutdown_connections 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc61c0a3cd0 0x7fc61c0a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc63006b1b0 0x7fc63019b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc630108a50 0x7fc63019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 --2- 172.21.15.105:0/4148880642 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc63006bb70 0x7fc63019b920 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.591 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 >> 172.21.15.105:0/4148880642 conn(0x7fc630077990 msgr2=0x7fc63010def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:35.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 shutdown_connections 2024-04-22T06:36:35.592 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.592+0000 7fc637c70700 1 -- 172.21.15.105:0/4148880642 wait complete. 2024-04-22T06:36:35.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 -- 172.21.15.105:0/4062173010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 msgr2=0x7f29a410e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 --2- 172.21.15.105:0/4062173010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a410e7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f299c00b0a0 tx=0x7f299c00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 -- 172.21.15.105:0/4062173010 shutdown_connections 2024-04-22T06:36:35.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 --2- 172.21.15.105:0/4062173010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 0x7f29a4107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 --2- 172.21.15.105:0/4062173010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a410e7e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 --2- 172.21.15.105:0/4062173010 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 0x7f29a4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 -- 172.21.15.105:0/4062173010 >> 172.21.15.105:0/4062173010 conn(0x7f29a40788b0 msgr2=0x7f29a4078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:35.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 -- 172.21.15.105:0/4062173010 shutdown_connections 2024-04-22T06:36:35.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.727+0000 7f29a98e6700 1 -- 172.21.15.105:0/4062173010 wait complete. 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.728+0000 7f29a98e6700 1 Processor -- start 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.728+0000 7f29a98e6700 1 -- start start 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 0x7f29a4197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 0x7f29a419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4111ae0 con 0x7f29a41022f0 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4111960 con 0x7f29a4107980 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29a4111c60 con 0x7f29a4101930 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48102/0 (socket says 172.21.15.105:48102) 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 -- 172.21.15.105:0/1549107493 learned_addr learned my addr 172.21.15.105:0/1549107493 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:35.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a2ffd700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 0x7f29a4197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a37fe700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 0x7f29a419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 -- 172.21.15.105:0/1549107493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 msgr2=0x7f29a4197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 0x7f29a4197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 -- 172.21.15.105:0/1549107493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 msgr2=0x7f29a419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 0x7f29a419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 -- 172.21.15.105:0/1549107493 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a419cf60 con 0x7f29a41022f0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a27fc700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f299800d980 tx=0x7f299800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998017070 con 0x7f29a41022f0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2998005020 con 0x7f29a41022f0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a419d250 con 0x7f29a41022f0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.729+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998009b50 con 0x7f29a41022f0 2024-04-22T06:36:35.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.730+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a41a4ad0 con 0x7f29a41022f0 2024-04-22T06:36:35.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.731+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a4069730 con 0x7f29a41022f0 2024-04-22T06:36:35.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.731+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f29980051d0 con 0x7f29a41022f0 2024-04-22T06:36:35.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.732+0000 7f29a88e4700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 0x7f298c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:35.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.732+0000 7f29a2ffd700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 0x7f298c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:35.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.732+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f29980d0370 con 0x7f29a41022f0 2024-04-22T06:36:35.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.733+0000 7f29a2ffd700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 0x7f298c0a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2994005950 tx=0x7f299400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:35.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.737+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2998020020 con 0x7f29a41022f0 2024-04-22T06:36:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:35 smithi105 ceph-mon[27916]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:35.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:35 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4148880642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:36:35.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.947+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f29a41098a0 con 0x7f29a41022f0 2024-04-22T06:36:35.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.948+0000 7f29a88e4700 1 -- 172.21.15.105:0/1549107493 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f299809cce0 con 0x7f29a41022f0 2024-04-22T06:36:35.946 INFO:teuthology.orchestra.run.smithi105.stdout:326417514525 2024-04-22T06:36:35.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 msgr2=0x7f298c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 0x7f298c0a6210 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2994005950 tx=0x7f299400a300 comp rx=0 tx=0).stop 2024-04-22T06:36:35.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 msgr2=0x7f29a4198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f299800d980 tx=0x7f299800dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 shutdown_connections 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f298c0a3d50 0x7f298c0a6210 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29a4107980 0x7f29a419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a4101930 0x7f29a4197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 --2- 172.21.15.105:0/1549107493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29a41022f0 0x7f29a4198490 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 >> 172.21.15.105:0/1549107493 conn(0x7f29a40788b0 msgr2=0x7f29a40ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 shutdown_connections 2024-04-22T06:36:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:35.950+0000 7f29a98e6700 1 -- 172.21.15.105:0/1549107493 wait complete. 2024-04-22T06:36:36.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:35 smithi112 ceph-mon[31236]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:36.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:35 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4148880642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:36:36.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:35 smithi177 ceph-mon[31346]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:36.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:35 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4148880642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T06:36:36.246 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:36.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 -- 172.21.15.105:0/3884975071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 msgr2=0x7f215c111b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 --2- 172.21.15.105:0/3884975071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c111b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f214c0099c0 tx=0x7f214c009cd0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 -- 172.21.15.105:0/3884975071 shutdown_connections 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 --2- 172.21.15.105:0/3884975071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c111b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 --2- 172.21.15.105:0/3884975071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c104eb0 0x7f215c105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 --2- 172.21.15.105:0/3884975071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c105860 0x7f215c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 -- 172.21.15.105:0/3884975071 >> 172.21.15.105:0/3884975071 conn(0x7f215c100720 msgr2=0x7f215c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 -- 172.21.15.105:0/3884975071 shutdown_connections 2024-04-22T06:36:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.259+0000 7f2162175700 1 -- 172.21.15.105:0/3884975071 wait complete. 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 Processor -- start 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 -- start start 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c104eb0 0x7f215c19b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c105860 0x7f215c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f215c114ea0 con 0x7f215c109f90 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f215c114d20 con 0x7f215c104eb0 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f2162175700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f215c115020 con 0x7f215c105860 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48120/0 (socket says 172.21.15.105:48120) 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 -- 172.21.15.105:0/2047389084 learned_addr learned my addr 172.21.15.105:0/2047389084 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215b7fe700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c104eb0 0x7f215c19b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215affd700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c105860 0x7f215c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 -- 172.21.15.105:0/2047389084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c105860 msgr2=0x7f215c19b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c105860 0x7f215c19b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 -- 172.21.15.105:0/2047389084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c104eb0 msgr2=0x7f215c19b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c104eb0 0x7f215c19b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.260+0000 7f215bfff700 1 -- 172.21.15.105:0/2047389084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c1a0340 con 0x7f215c109f90 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f215bfff700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f214c005910 tx=0x7f214c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c027070 con 0x7f215c109f90 2024-04-22T06:36:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215c1a05d0 con 0x7f215c109f90 2024-04-22T06:36:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f214c003f20 con 0x7f215c109f90 2024-04-22T06:36:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214c022ba0 con 0x7f215c109f90 2024-04-22T06:36:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.261+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215c1a7eb0 con 0x7f215c109f90 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.262+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215c069730 con 0x7f215c109f90 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.263+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f214c02a070 con 0x7f215c109f90 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.264+0000 7f2158ff9700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 0x7f21480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.264+0000 7f215b7fe700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 0x7f21480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.264+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f214c0d11d0 con 0x7f215c109f90 2024-04-22T06:36:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.264+0000 7f215b7fe700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 0x7f21480a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f215400a400 tx=0x7f215400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:36.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.272+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f214c013070 con 0x7f215c109f90 2024-04-22T06:36:36.352 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:36.352 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:36.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.499+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f215c02df80 con 0x7f215c109f90 2024-04-22T06:36:36.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.499+0000 7f2158ff9700 1 -- 172.21.15.105:0/2047389084 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f214c09dc90 con 0x7f215c109f90 2024-04-22T06:36:36.498 INFO:teuthology.orchestra.run.smithi105.stdout:300647710755 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 msgr2=0x7f21480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 0x7f21480a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f215400a400 tx=0x7f215400d040 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 msgr2=0x7f215c19fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f214c005910 tx=0x7f214c009580 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 shutdown_connections 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f21480a3cd0 0x7f21480a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f215c104eb0 0x7f215c19b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f215c109f90 0x7f215c19fc00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 --2- 172.21.15.105:0/2047389084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f215c105860 0x7f215c19b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 >> 172.21.15.105:0/2047389084 conn(0x7f215c100720 msgr2=0x7f215c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.501+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 shutdown_connections 2024-04-22T06:36:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.502+0000 7f2162175700 1 -- 172.21.15.105:0/2047389084 wait complete. 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 -- 172.21.15.105:0/3099635245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c108a50 msgr2=0x7fde3c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 --2- 172.21.15.105:0/3099635245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c108a50 0x7fde3c108e30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0099c0 tx=0x7fde2c009cd0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 -- 172.21.15.105:0/3099635245 shutdown_connections 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 --2- 172.21.15.105:0/3099635245 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 0x7fde3c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 --2- 172.21.15.105:0/3099635245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c108a50 0x7fde3c108e30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 --2- 172.21.15.105:0/3099635245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c06b1b0 0x7fde3c06b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.685+0000 7fde4084b700 1 -- 172.21.15.105:0/3099635245 >> 172.21.15.105:0/3099635245 conn(0x7fde3c077990 msgr2=0x7fde3c077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:36.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 -- 172.21.15.105:0/3099635245 shutdown_connections 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 -- 172.21.15.105:0/3099635245 wait complete. 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 Processor -- start 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 -- start start 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 0x7fde3c19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.686+0000 7fde4084b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c108a50 0x7fde3c19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde4084b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112c90 con 0x7fde3c06b1b0 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde4084b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112b10 con 0x7fde3c108a50 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde4084b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112e10 con 0x7fde3c06bb70 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48144/0 (socket says 172.21.15.105:48144) 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 -- 172.21.15.105:0/1923330603 learned_addr learned my addr 172.21.15.105:0/1923330603 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:36.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde39d9b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 0x7fde3c19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3ad9d700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c108a50 0x7fde3c19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 -- 172.21.15.105:0/1923330603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 msgr2=0x7fde3c19b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 0x7fde3c19b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 -- 172.21.15.105:0/1923330603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c108a50 msgr2=0x7fde3c19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c108a50 0x7fde3c19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 -- 172.21.15.105:0/1923330603 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1a0300 con 0x7fde3c06b1b0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde3a59c700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0098e0 tx=0x7fde2c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c027070 con 0x7fde3c06b1b0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c1a0590 con 0x7fde3c06b1b0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2c003f20 con 0x7fde3c06b1b0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c022d70 con 0x7fde3c06b1b0 2024-04-22T06:36:36.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.687+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a7e70 con 0x7fde3c06b1b0 2024-04-22T06:36:36.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.688+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3c069730 con 0x7fde3c06b1b0 2024-04-22T06:36:36.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.690+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fde2c02a070 con 0x7fde3c06b1b0 2024-04-22T06:36:36.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.690+0000 7fde2b7fe700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 0x7fde240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:36.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.690+0000 7fde39d9b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 0x7fde240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:36.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.690+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fde2c0d1930 con 0x7fde3c06b1b0 2024-04-22T06:36:36.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.691+0000 7fde39d9b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 0x7fde240a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde3c103a90 tx=0x7fde30009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:36.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.694+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde2c030070 con 0x7fde3c06b1b0 2024-04-22T06:36:36.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.904+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fde3c1a8040 con 0x7fde3c06b1b0 2024-04-22T06:36:36.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.905+0000 7fde2b7fe700 1 -- 172.21.15.105:0/1923330603 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde2c013070 con 0x7fde3c06b1b0 2024-04-22T06:36:36.903 INFO:teuthology.orchestra.run.smithi105.stdout:34359738447 2024-04-22T06:36:36.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 msgr2=0x7fde240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 0x7fde240a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde3c103a90 tx=0x7fde30009190 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 msgr2=0x7fde3c19b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0098e0 tx=0x7fde2c0057c0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 shutdown_connections 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fde240a3cd0 0x7fde240a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde3c108a50 0x7fde3c19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde3c06b1b0 0x7fde3c19b2f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 --2- 172.21.15.105:0/1923330603 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fde3c06bb70 0x7fde3c19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 >> 172.21.15.105:0/1923330603 conn(0x7fde3c077990 msgr2=0x7fde3c10dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 shutdown_connections 2024-04-22T06:36:36.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:36.907+0000 7fde4084b700 1 -- 172.21.15.105:0/1923330603 wait complete. 2024-04-22T06:36:37.145 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:36 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1549107493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:36:37.145 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:36 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2047389084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:36:37.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430255 got 206158430257 for osd.6 2024-04-22T06:36:37.174 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:36 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1549107493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:36:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:36 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2047389084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:36:37.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:36 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1549107493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-22T06:36:37.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:36 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2047389084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-22T06:36:37.827 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:37 smithi105 ceph-mon[27916]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:37.827 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:37 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1923330603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:36:37.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:37 smithi112 ceph-mon[31236]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:37 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1923330603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:36:38.147 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710751 got 300647710755 for osd.9 2024-04-22T06:36:38.147 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:38.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:37 smithi177 ceph-mon[31346]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-04-22T06:36:38.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:37 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1923330603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T06:36:38.756 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738447 for osd.0 2024-04-22T06:36:38.756 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:39.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514520 got 326417514525 for osd.10 2024-04-22T06:36:39.666 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.877+0000 7ff586bba700 1 -- 172.21.15.105:0/2965013453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580106cc0 msgr2=0x7ff58010e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.877+0000 7ff586bba700 1 --2- 172.21.15.105:0/2965013453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580106cc0 0x7ff58010e8d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff57800b0d0 tx=0x7ff57800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 -- 172.21.15.105:0/2965013453 shutdown_connections 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 --2- 172.21.15.105:0/2965013453 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff580102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 --2- 172.21.15.105:0/2965013453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580106cc0 0x7ff58010e8d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 --2- 172.21.15.105:0/2965013453 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580102d50 0x7ff5801065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 -- 172.21.15.105:0/2965013453 >> 172.21.15.105:0/2965013453 conn(0x7ff580077fb0 msgr2=0x7ff5800ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 -- 172.21.15.105:0/2965013453 shutdown_connections 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 -- 172.21.15.105:0/2965013453 wait complete. 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 Processor -- start 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 -- start start 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.878+0000 7ff586bba700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff586bba700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff586bba700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580106cc0 0x7ff58019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff586bba700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff580111bc0 con 0x7ff580102d50 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff586bba700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff580111a40 con 0x7ff5801023a0 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff586bba700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff580111d40 con 0x7ff580106cc0 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff584956700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff584956700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:32778/0 (socket says 172.21.15.105:32778) 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48176/0 (socket says 172.21.15.105:48176) 2024-04-22T06:36:39.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff584956700 1 -- 172.21.15.105:0/954023312 learned_addr learned my addr 172.21.15.105:0/954023312 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff585157700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580106cc0 0x7ff58019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 -- 172.21.15.105:0/954023312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580106cc0 msgr2=0x7ff58019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580106cc0 0x7ff58019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 -- 172.21.15.105:0/954023312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 msgr2=0x7ff58019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 -- 172.21.15.105:0/954023312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5801a0380 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff584956700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57ffff700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff5800752d0 tx=0x7ff56c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.879+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c016070 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.880+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5801a0670 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.880+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff56c00fd40 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.880+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00e3b0 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.880+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5801a7ef0 con 0x7ff580102d50 2024-04-22T06:36:39.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.881+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff580103150 con 0x7ff580102d50 2024-04-22T06:36:39.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.882+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff56c019070 con 0x7ff580102d50 2024-04-22T06:36:39.881 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.883+0000 7ff57dffb700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 0x7ff5700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:39.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.883+0000 7ff584956700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 0x7ff5700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:39.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.883+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7ff56c0cfd30 con 0x7ff580102d50 2024-04-22T06:36:39.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.884+0000 7ff584956700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 0x7ff5700a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff5740104e0 tx=0x7ff574007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:39.890 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:39 smithi105 ceph-mon[27916]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:36:39.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:39.892+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff56c09c760 con 0x7ff580102d50 2024-04-22T06:36:40.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.116+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff58002df80 con 0x7ff580102d50 2024-04-22T06:36:40.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.116+0000 7ff57dffb700 1 -- 172.21.15.105:0/954023312 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff56c09c060 con 0x7ff580102d50 2024-04-22T06:36:40.115 INFO:teuthology.orchestra.run.smithi105.stdout:77309411397 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.118+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 msgr2=0x7ff5700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.118+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 0x7ff5700a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff5740104e0 tx=0x7ff574007480 comp rx=0 tx=0).stop 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.118+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 msgr2=0x7ff58019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.118+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff5800752d0 tx=0x7ff56c00dc30 comp rx=0 tx=0).stop 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.118+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 shutdown_connections 2024-04-22T06:36:40.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7ff5700a3cd0 0x7ff5700a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5801023a0 0x7ff58019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff580106cc0 0x7ff58019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 --2- 172.21.15.105:0/954023312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff580102d50 0x7ff58019b8b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 >> 172.21.15.105:0/954023312 conn(0x7ff580077fb0 msgr2=0x7ff58010d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 shutdown_connections 2024-04-22T06:36:40.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.119+0000 7ff586bba700 1 -- 172.21.15.105:0/954023312 wait complete. 2024-04-22T06:36:40.133 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 -- 172.21.15.105:0/3718736770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0101030 msgr2=0x7fe7f0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 --2- 172.21.15.105:0/3718736770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0101030 0x7fe7f0101490 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009990 tx=0x7fe7e4009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 -- 172.21.15.105:0/3718736770 shutdown_connections 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 --2- 172.21.15.105:0/3718736770 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0108ad0 0x7fe7f0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 --2- 172.21.15.105:0/3718736770 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 --2- 172.21.15.105:0/3718736770 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0101030 0x7fe7f0101490 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 -- 172.21.15.105:0/3718736770 >> 172.21.15.105:0/3718736770 conn(0x7fe7f0077fb0 msgr2=0x7fe7f00ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 -- 172.21.15.105:0/3718736770 shutdown_connections 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.093+0000 7fe7f6421700 1 -- 172.21.15.105:0/3718736770 wait complete. 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.094+0000 7fe7f6421700 1 Processor -- start 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.094+0000 7fe7f6421700 1 -- start start 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 0x7fe7f01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:40.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0112be0 con 0x7fe7f0108ad0 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0112a60 con 0x7fe7f0101030 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f6421700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0112d60 con 0x7fe7f01019d0 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f49be700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f49be700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48198/0 (socket says 172.21.15.105:48198) 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:58368/0 (socket says 172.21.15.105:58368) 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f49be700 1 -- 172.21.15.105:0/498100008 learned_addr learned my addr 172.21.15.105:0/498100008 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7effff700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 0x7fe7f01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 -- 172.21.15.105:0/498100008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 msgr2=0x7fe7f01990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 0x7fe7f01990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 -- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 msgr2=0x7fe7f019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7ef7fe700 1 -- 172.21.15.105:0/498100008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f019e0e0 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7f49be700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.095+0000 7fe7effff700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 0x7fe7f01990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7ef7fe700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40059d0 tx=0x7fe7e40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4027070 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7e4022470 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f019e370 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4017650 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.096+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f0102140 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.097+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f0069730 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.098+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe7e402a070 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.099+0000 7fe7ed7fa700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 0x7fe7d80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.099+0000 7fe7effff700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 0x7fe7d80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.099+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fe7e40d10e0 con 0x7fe7f01019d0 2024-04-22T06:36:40.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.099+0000 7fe7effff700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 0x7fe7d80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009fd0 tx=0x7fe7e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:40.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.103+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7e4013070 con 0x7fe7f01019d0 2024-04-22T06:36:40.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:39 smithi112 ceph-mon[31236]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:36:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:39 smithi177 ceph-mon[31346]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:36:40.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.342+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe7f019a430 con 0x7fe7f01019d0 2024-04-22T06:36:40.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.342+0000 7fe7ed7fa700 1 -- 172.21.15.105:0/498100008 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe7e409daa0 con 0x7fe7f01019d0 2024-04-22T06:36:40.341 INFO:teuthology.orchestra.run.smithi105.stdout:146028888123 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 msgr2=0x7fe7d80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 0x7fe7d80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009fd0 tx=0x7fe7e0009260 comp rx=0 tx=0).stop 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 msgr2=0x7fe7f0199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe7e40059d0 tx=0x7fe7e40047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 shutdown_connections 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe7d80a3cd0 0x7fe7d80a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe7f0101030 0x7fe7f01990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7f0108ad0 0x7fe7f019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 --2- 172.21.15.105:0/498100008 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe7f01019d0 0x7fe7f0199610 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:40.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.344+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 >> 172.21.15.105:0/498100008 conn(0x7fe7f0077fb0 msgr2=0x7fe7f00ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:40.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.345+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 shutdown_connections 2024-04-22T06:36:40.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:40.345+0000 7fe7f6421700 1 -- 172.21.15.105:0/498100008 wait complete. 2024-04-22T06:36:40.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:40 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:40.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/954023312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:36:40.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/498100008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:36:40.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:40 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:40.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/954023312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:36:40.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/498100008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:36:41.041 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:40 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:36:41.041 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/954023312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T06:36:41.042 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/498100008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T06:36:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:41 smithi105 ceph-mon[27916]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:41 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:41.981 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888119 got 146028888123 for osd.4 2024-04-22T06:36:41.981 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:41 smithi112 ceph-mon[31236]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:41 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:41 smithi177 ceph-mon[31346]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:36:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:41 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:43.652 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:44.002 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:43 smithi105 ceph-mon[27916]: pgmap v318: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:43 smithi112 ceph-mon[31236]: pgmap v318: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:43 smithi177 ceph-mon[31346]: pgmap v318: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:36:44.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 -- 172.21.15.105:0/1231697829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c102300 msgr2=0x7eff8c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:44.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 --2- 172.21.15.105:0/1231697829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c102300 0x7eff8c10e7f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7eff8400b0a0 tx=0x7eff8400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 -- 172.21.15.105:0/1231697829 shutdown_connections 2024-04-22T06:36:44.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 --2- 172.21.15.105:0/1231697829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c102300 0x7eff8c10e7f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 --2- 172.21.15.105:0/1231697829 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c107990 0x7eff8c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 --2- 172.21.15.105:0/1231697829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c101940 0x7eff8c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 -- 172.21.15.105:0/1231697829 >> 172.21.15.105:0/1231697829 conn(0x7eff8c0788f0 msgr2=0x7eff8c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 -- 172.21.15.105:0/1231697829 shutdown_connections 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.645+0000 7eff91f06700 1 -- 172.21.15.105:0/1231697829 wait complete. 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 Processor -- start 2024-04-22T06:36:44.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 -- start start 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c101940 0x7eff8c199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c102300 0x7eff8c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c111b60 con 0x7eff8c107990 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c1119e0 con 0x7eff8c102300 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff91f06700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c111ce0 con 0x7eff8c101940 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff8bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff8bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42898/0 (socket says 172.21.15.105:42898) 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.646+0000 7eff8bfff700 1 -- 172.21.15.105:0/1420886797 learned_addr learned my addr 172.21.15.105:0/1420886797 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:44.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8b7fe700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c101940 0x7eff8c199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8affd700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c102300 0x7eff8c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 -- 172.21.15.105:0/1420886797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c101940 msgr2=0x7eff8c199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c101940 0x7eff8c199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 -- 172.21.15.105:0/1420886797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c102300 msgr2=0x7eff8c19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c102300 0x7eff8c19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 -- 172.21.15.105:0/1420886797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c1a54a0 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff8bfff700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff840062a0 tx=0x7eff84004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84021070 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff84007d00 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8401d450 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8c1a5670 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.647+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8c1a5b70 con 0x7eff8c107990 2024-04-22T06:36:44.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.648+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8c109eb0 con 0x7eff8c107990 2024-04-22T06:36:44.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.649+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7eff84004020 con 0x7eff8c107990 2024-04-22T06:36:44.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.650+0000 7eff88ff9700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 0x7eff740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:44.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.650+0000 7eff8b7fe700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 0x7eff740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:44.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.650+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7eff8400f020 con 0x7eff8c107990 2024-04-22T06:36:44.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.650+0000 7eff8b7fe700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 0x7eff740a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:44.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.655+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff8402a020 con 0x7eff8c107990 2024-04-22T06:36:44.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.868+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7eff8c06ad30 con 0x7eff8c107990 2024-04-22T06:36:44.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.868+0000 7eff88ff9700 1 -- 172.21.15.105:0/1420886797 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7eff8409e520 con 0x7eff8c107990 2024-04-22T06:36:44.867 INFO:teuthology.orchestra.run.smithi105.stdout:266287972393 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 msgr2=0x7eff740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 0x7eff740a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80008040 comp rx=0 tx=0).stop 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 msgr2=0x7eff8c1a4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7eff840062a0 tx=0x7eff84004ed0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 shutdown_connections 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7eff740a3d50 0x7eff740a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff8c107990 0x7eff8c1a4f60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7eff8c101940 0x7eff8c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 --2- 172.21.15.105:0/1420886797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff8c102300 0x7eff8c19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:44.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 >> 172.21.15.105:0/1420886797 conn(0x7eff8c0788f0 msgr2=0x7eff8c0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:44.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 shutdown_connections 2024-04-22T06:36:44.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:44.870+0000 7eff91f06700 1 -- 172.21.15.105:0/1420886797 wait complete. 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:36:44.880 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:44 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-22T06:36:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T06:36:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:36:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T06:36:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-22T06:36:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:36:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:44 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-22T06:36:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:44 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-22T06:36:46.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: pgmap v319: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:36:46.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:45 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1420886797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: pgmap v319: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:36:46.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:45 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1420886797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: pgmap v319: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: osdmap e90: 12 total, 12 up, 12 in 2024-04-22T06:36:46.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:45 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1420886797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-22T06:36:47.101 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:46 smithi105 ceph-mon[27916]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:36:47.101 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:46 smithi105 ceph-mon[27916]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-04-22T06:36:47.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411391 got 77309411397 for osd.2 2024-04-22T06:36:47.132 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:47.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:46 smithi112 ceph-mon[31236]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:36:47.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:46 smithi112 ceph-mon[31236]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-04-22T06:36:47.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:46 smithi177 ceph-mon[31346]: osdmap e91: 12 total, 12 up, 12 in 2024-04-22T06:36:47.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:46 smithi177 ceph-mon[31346]: pgmap v322: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-04-22T06:36:47.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 -- 172.21.15.105:0/4155568098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 msgr2=0x7fb1cc10f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/4155568098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc10f830 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400b0a0 tx=0x7fb1c400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 -- 172.21.15.105:0/4155568098 shutdown_connections 2024-04-22T06:36:47.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/4155568098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc10f830 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/4155568098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 0x7fb1cc107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/4155568098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 0x7fb1cc101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.543+0000 7fb1d1efd700 1 -- 172.21.15.105:0/4155568098 >> 172.21.15.105:0/4155568098 conn(0x7fb1cc076c60 msgr2=0x7fb1cc077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:47.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.544+0000 7fb1d1efd700 1 -- 172.21.15.105:0/4155568098 shutdown_connections 2024-04-22T06:36:47.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.544+0000 7fb1d1efd700 1 -- 172.21.15.105:0/4155568098 wait complete. 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.544+0000 7fb1d1efd700 1 Processor -- start 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.544+0000 7fb1d1efd700 1 -- start start 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.544+0000 7fb1d1efd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 0x7fb1cc199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 0x7fb1cc19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112cb0 con 0x7fb1cc101dc0 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112b30 con 0x7fb1cc101400 2024-04-22T06:36:47.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1cc112e30 con 0x7fb1cc107b20 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42918/0 (socket says 172.21.15.105:42918) 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 -- 172.21.15.105:0/2702847796 learned_addr learned my addr 172.21.15.105:0/2702847796 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1cbfff700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 0x7fb1cc19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 0x7fb1cc199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 -- 172.21.15.105:0/2702847796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 msgr2=0x7fb1cc19d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 0x7fb1cc19d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 -- 172.21.15.105:0/2702847796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 msgr2=0x7fb1cc199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 0x7fb1cc199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 -- 172.21.15.105:0/2702847796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1cc19e040 con 0x7fb1cc101dc0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1caffd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000d980 tx=0x7fb1c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0016070 con 0x7fb1cc101dc0 2024-04-22T06:36:47.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1cc19e330 con 0x7fb1cc101dc0 2024-04-22T06:36:47.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0004830 con 0x7fb1cc101dc0 2024-04-22T06:36:47.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c00054c0 con 0x7fb1cc101dc0 2024-04-22T06:36:47.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.545+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1cc1a5bb0 con 0x7fb1cc101dc0 2024-04-22T06:36:47.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.546+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc069730 con 0x7fb1cc101dc0 2024-04-22T06:36:47.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.547+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fb1c0004220 con 0x7fb1cc101dc0 2024-04-22T06:36:47.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.548+0000 7fb1c8ff9700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 0x7fb1b40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.548+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 0x7fb1b40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.548+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fb1c00d07b0 con 0x7fb1cc101dc0 2024-04-22T06:36:47.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.548+0000 7fb1cb7fe700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 0x7fb1b40a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0781e0 tx=0x7fb1bc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:47.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.552+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1c001f020 con 0x7fb1cc101dc0 2024-04-22T06:36:47.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.757+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb1cc02df80 con 0x7fb1cc101dc0 2024-04-22T06:36:47.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.757+0000 7fb1c8ff9700 1 -- 172.21.15.105:0/2702847796 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb1c009d130 con 0x7fb1cc101dc0 2024-04-22T06:36:47.756 INFO:teuthology.orchestra.run.smithi105.stdout:55834574923 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 msgr2=0x7fb1b40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 0x7fb1b40a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0781e0 tx=0x7fb1bc005ee0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 msgr2=0x7fb1cc1995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000d980 tx=0x7fb1c000dc90 comp rx=0 tx=0).stop 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 shutdown_connections 2024-04-22T06:36:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb1b40a3d50 0x7fb1b40a6210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1cc101400 0x7fb1cc199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1cc107b20 0x7fb1cc19d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 --2- 172.21.15.105:0/2702847796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1cc101dc0 0x7fb1cc1995c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 >> 172.21.15.105:0/2702847796 conn(0x7fb1cc076c60 msgr2=0x7fb1cc0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 shutdown_connections 2024-04-22T06:36:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.759+0000 7fb1d1efd700 1 -- 172.21.15.105:0/2702847796 wait complete. 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 -- 172.21.15.105:0/1017600729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 msgr2=0x7f8c1010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 --2- 172.21.15.105:0/1017600729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c1010f910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800b0a0 tx=0x7f8c0800b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 -- 172.21.15.105:0/1017600729 shutdown_connections 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 --2- 172.21.15.105:0/1017600729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c10108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 --2- 172.21.15.105:0/1017600729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c1010f910 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 --2- 172.21.15.105:0/1017600729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c10101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 -- 172.21.15.105:0/1017600729 >> 172.21.15.105:0/1017600729 conn(0x7f8c10077fb0 msgr2=0x7f8c100ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 -- 172.21.15.105:0/1017600729 shutdown_connections 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.952+0000 7f8c1743a700 1 -- 172.21.15.105:0/1017600729 wait complete. 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 Processor -- start 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 -- start start 2024-04-22T06:36:47.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c101992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c1019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8c10112cc0 con 0x7f8c101019d0 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8c10112b40 con 0x7f8c10108ad0 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c1743a700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8c10112e40 con 0x7f8c10101030 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42936/0 (socket says 172.21.15.105:42936) 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 -- 172.21.15.105:0/313301695 learned_addr learned my addr 172.21.15.105:0/313301695 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c151d6700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c101992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c159d7700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c1019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 -- 172.21.15.105:0/313301695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 msgr2=0x7f8c101992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c101992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 -- 172.21.15.105:0/313301695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 msgr2=0x7f8c1019db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c1019db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 -- 172.21.15.105:0/313301695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1019e2d0 con 0x7f8c101019d0 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c159d7700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c1019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c151d6700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c101992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.953+0000 7f8c149d5700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000b530 tx=0x7f8c0000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:47.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.954+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00015020 con 0x7f8c101019d0 2024-04-22T06:36:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.954+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1019e5c0 con 0x7f8c101019d0 2024-04-22T06:36:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.954+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c00011ce0 con 0x7f8c101019d0 2024-04-22T06:36:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.954+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00010d20 con 0x7f8c101019d0 2024-04-22T06:36:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.954+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c10102140 con 0x7f8c101019d0 2024-04-22T06:36:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.955+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c10069730 con 0x7f8c101019d0 2024-04-22T06:36:47.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.956+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8c00018020 con 0x7f8c101019d0 2024-04-22T06:36:47.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.956+0000 7f8c067fc700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 0x7f8bfc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:47.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.956+0000 7f8c151d6700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 0x7f8bfc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:47.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.956+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f8c000d0140 con 0x7f8c101019d0 2024-04-22T06:36:47.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.957+0000 7f8c151d6700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 0x7f8bfc0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0099d0 tx=0x7f8c0c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:47.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:47.960+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c0001f020 con 0x7f8c101019d0 2024-04-22T06:36:48.057 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:47 smithi105 ceph-mon[27916]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T06:36:48.057 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2702847796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:36:48.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.161+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8c1002df80 con 0x7f8c101019d0 2024-04-22T06:36:48.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.161+0000 7f8c067fc700 1 -- 172.21.15.105:0/313301695 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c0009cbd0 con 0x7f8c101019d0 2024-04-22T06:36:48.160 INFO:teuthology.orchestra.run.smithi105.stdout:227633266734 2024-04-22T06:36:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.163+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 msgr2=0x7f8bfc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.163+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 0x7f8bfc0a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0099d0 tx=0x7f8c0c008040 comp rx=0 tx=0).stop 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.163+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 msgr2=0x7f8c10199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.163+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000b530 tx=0x7f8c0000b8f0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 shutdown_connections 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8bfc0a3cd0 0x7f8bfc0a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c10108ad0 0x7f8c1019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c10101030 0x7f8c101992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 --2- 172.21.15.105:0/313301695 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8c101019d0 0x7f8c10199800 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 >> 172.21.15.105:0/313301695 conn(0x7f8c10077fb0 msgr2=0x7f8c100ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 shutdown_connections 2024-04-22T06:36:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.164+0000 7f8c1743a700 1 -- 172.21.15.105:0/313301695 wait complete. 2024-04-22T06:36:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:47 smithi112 ceph-mon[31236]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T06:36:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2702847796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:36:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:47 smithi177 ceph-mon[31346]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-22T06:36:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2702847796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 -- 172.21.15.105:0/3435272209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 msgr2=0x7f39ac10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/3435272209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac10e7f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f39a400b0a0 tx=0x7f39a400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 -- 172.21.15.105:0/3435272209 shutdown_connections 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/3435272209 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac107990 0x7f39ac107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/3435272209 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac10e7f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/3435272209 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac101940 0x7f39ac101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 -- 172.21.15.105:0/3435272209 >> 172.21.15.105:0/3435272209 conn(0x7f39ac0788f0 msgr2=0x7f39ac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 -- 172.21.15.105:0/3435272209 shutdown_connections 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.515+0000 7f39b3dbe700 1 -- 172.21.15.105:0/3435272209 wait complete. 2024-04-22T06:36:48.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 Processor -- start 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 -- start start 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac101940 0x7f39ac199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac107990 0x7f39ac1a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac111b50 con 0x7f39ac102300 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac1119d0 con 0x7f39ac101940 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b3dbe700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac111cd0 con 0x7f39ac107990 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b1359700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b1359700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42950/0 (socket says 172.21.15.105:42950) 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b1359700 1 -- 172.21.15.105:0/2190113934 learned_addr learned my addr 172.21.15.105:0/2190113934 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.516+0000 7f39b1b5a700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac101940 0x7f39ac199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:48.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 -- 172.21.15.105:0/2190113934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac107990 msgr2=0x7f39ac1a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b235b700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac107990 0x7f39ac1a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac107990 0x7f39ac1a5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 -- 172.21.15.105:0/2190113934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac101940 msgr2=0x7f39ac199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac101940 0x7f39ac199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 -- 172.21.15.105:0/2190113934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac1a5550 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b1359700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f39a800b530 tx=0x7f39a800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8015020 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39a8011ce0 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac1a5720 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8010e40 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.517+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ac1a5c10 con 0x7f39ac102300 2024-04-22T06:36:48.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.518+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ac069730 con 0x7f39ac102300 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.520+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f39a8018020 con 0x7f39ac102300 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.520+0000 7f39a2ffd700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 0x7f39980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.520+0000 7f39b1b5a700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 0x7f39980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.520+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f39a80d01a0 con 0x7f39ac102300 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.521+0000 7f39b1b5a700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 0x7f39980a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f399c005950 tx=0x7f399c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:48.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.524+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39a801f020 con 0x7f39ac102300 2024-04-22T06:36:48.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.730+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f39ac06ace0 con 0x7f39ac102300 2024-04-22T06:36:48.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.730+0000 7f39a2ffd700 1 -- 172.21.15.105:0/2190113934 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f39a809cc30 con 0x7f39ac102300 2024-04-22T06:36:48.729 INFO:teuthology.orchestra.run.smithi105.stdout:171798691895 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 msgr2=0x7f39980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 0x7f39980a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f399c005950 tx=0x7f399c00b410 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 msgr2=0x7f39ac19a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f39a800b530 tx=0x7f39a800b840 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 shutdown_connections 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f39980a3cd0 0x7f39980a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39ac101940 0x7f39ac199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f39ac107990 0x7f39ac1a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 --2- 172.21.15.105:0/2190113934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f39ac102300 0x7f39ac19a4d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 >> 172.21.15.105:0/2190113934 conn(0x7f39ac0788f0 msgr2=0x7f39ac0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 shutdown_connections 2024-04-22T06:36:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:48.732+0000 7f39b3dbe700 1 -- 172.21.15.105:0/2190113934 wait complete. 2024-04-22T06:36:48.965 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574919 got 55834574923 for osd.1 2024-04-22T06:36:48.965 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/313301695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:36:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:48 smithi105 ceph-mon[27916]: pgmap v323: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2190113934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:36:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/313301695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:36:49.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:48 smithi112 ceph-mon[31236]: pgmap v323: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:49.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2190113934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:36:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/313301695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T06:36:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:48 smithi177 ceph-mon[31346]: pgmap v323: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2190113934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T06:36:49.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972386 got 266287972393 for osd.8 2024-04-22T06:36:49.520 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:50.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266730 got 227633266734 for osd.7 2024-04-22T06:36:50.301 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 -- 172.21.15.105:0/3961743659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06b200 msgr2=0x7f779c06b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 --2- 172.21.15.105:0/3961743659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06b200 0x7f779c06b660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f778c009990 tx=0x7f778c009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 -- 172.21.15.105:0/3961743659 shutdown_connections 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 --2- 172.21.15.105:0/3961743659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06bba0 0x7f779c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 --2- 172.21.15.105:0/3961743659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 0x7f779c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 --2- 172.21.15.105:0/3961743659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06b200 0x7f779c06b660 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 -- 172.21.15.105:0/3961743659 >> 172.21.15.105:0/3961743659 conn(0x7f779c0788f0 msgr2=0x7f779c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 -- 172.21.15.105:0/3961743659 shutdown_connections 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 -- 172.21.15.105:0/3961743659 wait complete. 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.539+0000 7f77a25a8700 1 Processor -- start 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 -- start start 2024-04-22T06:36:50.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06b200 0x7f779c199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 0x7f779c19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f779c112cb0 con 0x7f779c06bba0 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f779c112b30 con 0x7f779c06b200 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a25a8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f779c112e30 con 0x7f779c108a30 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f779b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f779b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42970/0 (socket says 172.21.15.105:42970) 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f779b7fe700 1 -- 172.21.15.105:0/4011461396 learned_addr learned my addr 172.21.15.105:0/4011461396 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f779bfff700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06b200 0x7f779c199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.540+0000 7f77a0b45700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 0x7f779c19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 -- 172.21.15.105:0/4011461396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 msgr2=0x7f779c19db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 0x7f779c19db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 -- 172.21.15.105:0/4011461396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06b200 msgr2=0x7f779c199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06b200 0x7f779c199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 -- 172.21.15.105:0/4011461396 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c19e290 con 0x7f779c06bba0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f779b7fe700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f778c000c00 tx=0x7f778c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c003c20 con 0x7f779c06bba0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779c19e520 con 0x7f779c06bba0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f779c101940 con 0x7f779c06bba0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f778c01f4d0 con 0x7f779c06bba0 2024-04-22T06:36:50.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.541+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778c00f6c0 con 0x7f779c06bba0 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.543+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7788005330 con 0x7f779c06bba0 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.543+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f778c00f920 con 0x7f779c06bba0 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.544+0000 7f77997fa700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 0x7f77840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.544+0000 7f779bfff700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 0x7f77840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.544+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f778c0d56c0 con 0x7f779c06bba0 2024-04-22T06:36:50.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.545+0000 7f779bfff700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 0x7f77840a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7790009a10 tx=0x7f7790009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:50.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.549+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f778c013070 con 0x7f779c06bba0 2024-04-22T06:36:50.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.759+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f77880058f0 con 0x7f779c06bba0 2024-04-22T06:36:50.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.759+0000 7f77997fa700 1 -- 172.21.15.105:0/4011461396 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f778c0a21e0 con 0x7f779c06bba0 2024-04-22T06:36:50.758 INFO:teuthology.orchestra.run.smithi105.stdout:111669149761 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 msgr2=0x7f77840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 0x7f77840a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7790009a10 tx=0x7f7790009190 comp rx=0 tx=0).stop 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 msgr2=0x7f779c1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f778c000c00 tx=0x7f778c0047b0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 shutdown_connections 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.761+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f77840a3d50 0x7f77840a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f779c06b200 0x7f779c199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f779c108a30 0x7f779c19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 --2- 172.21.15.105:0/4011461396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f779c06bba0 0x7f779c1997c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:50.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 >> 172.21.15.105:0/4011461396 conn(0x7f779c0788f0 msgr2=0x7f779c0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:50.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 shutdown_connections 2024-04-22T06:36:50.761 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:50.762+0000 7f77a25a8700 1 -- 172.21.15.105:0/4011461396 wait complete. 2024-04-22T06:36:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:50 smithi105 ceph-mon[27916]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T06:36:50.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:50 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:36:50.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691890 got 171798691895 for osd.5 2024-04-22T06:36:50.973 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:50 smithi177 ceph-mon[31346]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T06:36:50.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:50 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:36:51.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:50 smithi112 ceph-mon[31236]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-22T06:36:51.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:50 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:36:51.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149755 got 111669149761 for osd.3 2024-04-22T06:36:51.625 DEBUG:teuthology.parallel:result is None 2024-04-22T06:36:51.625 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T06:36:51.625 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:36:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:51 smithi105 ceph-mon[27916]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:51.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4011461396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:51 smithi177 ceph-mon[31346]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:51.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:51 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4011461396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:36:52.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:51 smithi112 ceph-mon[31236]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:52.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:51 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4011461396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T06:36:52.444 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:53.863 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:53 smithi105 ceph-mon[27916]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:54.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 -- 172.21.15.105:0/3789871036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4108a50 msgr2=0x7fa9b4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 --2- 172.21.15.105:0/3789871036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4108a50 0x7fa9b4108e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009990 tx=0x7fa9b0009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 -- 172.21.15.105:0/3789871036 shutdown_connections 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 --2- 172.21.15.105:0/3789871036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4103450 0x7fa9b410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 --2- 172.21.15.105:0/3789871036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4108a50 0x7fa9b4108e30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 --2- 172.21.15.105:0/3789871036 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 0x7fa9b4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 -- 172.21.15.105:0/3789871036 >> 172.21.15.105:0/3789871036 conn(0x7fa9b4079ba0 msgr2=0x7fa9b4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 -- 172.21.15.105:0/3789871036 shutdown_connections 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.149+0000 7fa9bb295700 1 -- 172.21.15.105:0/3789871036 wait complete. 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 Processor -- start 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 -- start start 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 0x7fa9b41992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4108a50 0x7fa9b419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b4112ce0 con 0x7fa9b4103450 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b4112b60 con 0x7fa9b4108a50 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9bb295700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b4112e60 con 0x7fa9b4102a90 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9b8830700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9b8830700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42986/0 (socket says 172.21.15.105:42986) 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.150+0000 7fa9b8830700 1 -- 172.21.15.105:0/4129065223 learned_addr learned my addr 172.21.15.105:0/4129065223 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b9031700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 0x7fa9b41992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 -- 172.21.15.105:0/4129065223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 msgr2=0x7fa9b41992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b9832700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4108a50 0x7fa9b419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 0x7fa9b41992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 -- 172.21.15.105:0/4129065223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4108a50 msgr2=0x7fa9b419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4108a50 0x7fa9b419db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 -- 172.21.15.105:0/4129065223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b419e260 con 0x7fa9b4103450 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9b8830700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400b530 tx=0x7fa9a400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:54.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4015020 con 0x7fa9b4103450 2024-04-22T06:36:54.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a4011ce0 con 0x7fa9b4103450 2024-04-22T06:36:54.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b419e550 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4010d20 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.151+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b41a5dd0 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.152+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b4104180 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.153+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa9a4018020 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.154+0000 7fa9aa7fc700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 0x7fa9a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.154+0000 7fa9b9031700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 0x7fa9a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.154+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fa9a40cf9b0 con 0x7fa9b4103450 2024-04-22T06:36:54.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.154+0000 7fa9b9031700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 0x7fa9a00a6070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0005cd0 tx=0x7fa9b0005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:54.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.158+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9a401f030 con 0x7fa9b4103450 2024-04-22T06:36:54.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:53 smithi112 ceph-mon[31236]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:54.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:53 smithi177 ceph-mon[31346]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:54.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.365+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9b419a5b0 con 0x7fa9a00a3bb0 2024-04-22T06:36:54.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.373+0000 7fa9aa7fc700 1 -- 172.21.15.105:0/4129065223 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258774 (secure 0 0 0) 0x7fa9b419a5b0 con 0x7fa9a00a3bb0 2024-04-22T06:36:54.377 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 msgr2=0x7fa9a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 0x7fa9a00a6070 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0005cd0 tx=0x7fa9b0005c40 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 msgr2=0x7fa9b4199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400b530 tx=0x7fa9a400b8f0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 shutdown_connections 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa9a00a3bb0 0x7fa9a00a6070 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa9b4108a50 0x7fa9b419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa9b4102a90 0x7fa9b41992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 --2- 172.21.15.105:0/4129065223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9b4103450 0x7fa9b4199830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 >> 172.21.15.105:0/4129065223 conn(0x7fa9b4079ba0 msgr2=0x7fa9b4101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 shutdown_connections 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:54.376+0000 7fa9bb295700 1 -- 172.21.15.105:0/4129065223 wait complete. 2024-04-22T06:36:54.379 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:36:55.036 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-22T06:36:52.637295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656880,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":816,"num_read_kb":534,"num_write":418,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":292,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375536,"kb_used_data":13836,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566288,"statfs":{"total":1151940427776,"available":1151555878912,"internally_reserved":0,"allocated":14168064,"data_stored":7675716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001950"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783094+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:36:45.783094+0000","last_peered":"2024-04-22T06:36:45.783094+0000","last_clean":"2024-04-22T06:36:45.783094+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:36:45.783094+0000","last_undegraded":"2024-04-22T06:36:45.783094+0000","last_fullsized":"2024-04-22T06:36:45.783094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032004+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:36:46.032004+0000","last_peered":"2024-04-22T06:36:46.032004+0000","last_clean":"2024-04-22T06:36:46.032004+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:36:46.032004+0000","last_undegraded":"2024-04-22T06:36:46.032004+0000","last_fullsized":"2024-04-22T06:36:46.032004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031923+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:36:46.031923+0000","last_peered":"2024-04-22T06:36:46.031923+0000","last_clean":"2024-04-22T06:36:46.031923+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:36:46.031923+0000","last_undegraded":"2024-04-22T06:36:46.031923+0000","last_fullsized":"2024-04-22T06:36:46.031923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032350+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:36:46.032350+0000","last_peered":"2024-04-22T06:36:46.032350+0000","last_clean":"2024-04-22T06:36:46.032350+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:36:46.032350+0000","last_undegraded":"2024-04-22T06:36:46.032350+0000","last_fullsized":"2024-04-22T06:36:46.032350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785549+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:36:45.785549+0000","last_peered":"2024-04-22T06:36:45.785549+0000","last_clean":"2024-04-22T06:36:45.785549+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:36:45.785549+0000","last_undegraded":"2024-04-22T06:36:45.785549+0000","last_fullsized":"2024-04-22T06:36:45.785549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785686+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:36:45.785686+0000","last_peered":"2024-04-22T06:36:45.785686+0000","last_clean":"2024-04-22T06:36:45.785686+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:36:45.785686+0000","last_undegraded":"2024-04-22T06:36:45.785686+0000","last_fullsized":"2024-04-22T06:36:45.785686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785109+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:36:45.785109+0000","last_peered":"2024-04-22T06:36:45.785109+0000","last_clean":"2024-04-22T06:36:45.785109+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:36:45.785109+0000","last_undegraded":"2024-04-22T06:36:45.785109+0000","last_fullsized":"2024-04-22T06:36:45.785109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784799+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:36:45.784799+0000","last_peered":"2024-04-22T06:36:45.784799+0000","last_clean":"2024-04-22T06:36:45.784799+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:36:45.784799+0000","last_undegraded":"2024-04-22T06:36:45.784799+0000","last_fullsized":"2024-04-22T06:36:45.784799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.781064+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:36:45.781064+0000","last_peered":"2024-04-22T06:36:45.781064+0000","last_clean":"2024-04-22T06:36:45.781064+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:36:45.781064+0000","last_undegraded":"2024-04-22T06:36:45.781064+0000","last_fullsized":"2024-04-22T06:36:45.781064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786201+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:36:45.786201+0000","last_peered":"2024-04-22T06:36:45.786201+0000","last_clean":"2024-04-22T06:36:45.786201+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:36:45.786201+0000","last_undegraded":"2024-04-22T06:36:45.786201+0000","last_fullsized":"2024-04-22T06:36:45.786201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785024+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:36:45.785024+0000","last_peered":"2024-04-22T06:36:45.785024+0000","last_clean":"2024-04-22T06:36:45.785024+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:36:45.785024+0000","last_undegraded":"2024-04-22T06:36:45.785024+0000","last_fullsized":"2024-04-22T06:36:45.785024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.788195+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:36:45.788195+0000","last_peered":"2024-04-22T06:36:45.788195+0000","last_clean":"2024-04-22T06:36:45.788195+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:36:45.788195+0000","last_undegraded":"2024-04-22T06:36:45.788195+0000","last_fullsized":"2024-04-22T06:36:45.788195+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782412+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:36:45.782412+0000","last_peered":"2024-04-22T06:36:45.782412+0000","last_clean":"2024-04-22T06:36:45.782412+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:36:45.782412+0000","last_undegraded":"2024-04-22T06:36:45.782412+0000","last_fullsized":"2024-04-22T06:36:45.782412+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031721+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:36:46.031721+0000","last_peered":"2024-04-22T06:36:46.031721+0000","last_clean":"2024-04-22T06:36:46.031721+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:36:46.031721+0000","last_undegraded":"2024-04-22T06:36:46.031721+0000","last_fullsized":"2024-04-22T06:36:46.031721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031655+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:36:46.031655+0000","last_peered":"2024-04-22T06:36:46.031655+0000","last_clean":"2024-04-22T06:36:46.031655+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:36:46.031655+0000","last_undegraded":"2024-04-22T06:36:46.031655+0000","last_fullsized":"2024-04-22T06:36:46.031655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780414+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:36:45.780414+0000","last_peered":"2024-04-22T06:36:45.780414+0000","last_clean":"2024-04-22T06:36:45.780414+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:36:45.780414+0000","last_undegraded":"2024-04-22T06:36:45.780414+0000","last_fullsized":"2024-04-22T06:36:45.780414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783262+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:36:45.783262+0000","last_peered":"2024-04-22T06:36:45.783262+0000","last_clean":"2024-04-22T06:36:45.783262+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:36:45.783262+0000","last_undegraded":"2024-04-22T06:36:45.783262+0000","last_fullsized":"2024-04-22T06:36:45.783262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780172+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:36:45.780172+0000","last_peered":"2024-04-22T06:36:45.780172+0000","last_clean":"2024-04-22T06:36:45.780172+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:36:45.780172+0000","last_undegraded":"2024-04-22T06:36:45.780172+0000","last_fullsized":"2024-04-22T06:36:45.780172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780471+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:36:45.780471+0000","last_peered":"2024-04-22T06:36:45.780471+0000","last_clean":"2024-04-22T06:36:45.780471+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:36:45.780471+0000","last_undegraded":"2024-04-22T06:36:45.780471+0000","last_fullsized":"2024-04-22T06:36:45.780471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:47.083650+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:36:47.083650+0000","last_peered":"2024-04-22T06:36:47.083650+0000","last_clean":"2024-04-22T06:36:47.083650+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:36:47.083650+0000","last_undegraded":"2024-04-22T06:36:47.083650+0000","last_fullsized":"2024-04-22T06:36:47.083650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":63,"num_read_kb":63,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785601+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:36:45.785601+0000","last_peered":"2024-04-22T06:36:45.785601+0000","last_clean":"2024-04-22T06:36:45.785601+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:36:45.785601+0000","last_undegraded":"2024-04-22T06:36:45.785601+0000","last_fullsized":"2024-04-22T06:36:45.785601+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785737+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:36:45.785737+0000","last_peered":"2024-04-22T06:36:45.785737+0000","last_clean":"2024-04-22T06:36:45.785737+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:36:45.785737+0000","last_undegraded":"2024-04-22T06:36:45.785737+0000","last_fullsized":"2024-04-22T06:36:45.785737+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032036+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:36:46.032036+0000","last_peered":"2024-04-22T06:36:46.032036+0000","last_clean":"2024-04-22T06:36:46.032036+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:36:46.032036+0000","last_undegraded":"2024-04-22T06:36:46.032036+0000","last_fullsized":"2024-04-22T06:36:46.032036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.799058+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:36:45.799058+0000","last_peered":"2024-04-22T06:36:45.799058+0000","last_clean":"2024-04-22T06:36:45.799058+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:36:45.799058+0000","last_undegraded":"2024-04-22T06:36:45.799058+0000","last_fullsized":"2024-04-22T06:36:45.799058+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780711+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:36:45.780711+0000","last_peered":"2024-04-22T06:36:45.780711+0000","last_clean":"2024-04-22T06:36:45.780711+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:36:45.780711+0000","last_undegraded":"2024-04-22T06:36:45.780711+0000","last_fullsized":"2024-04-22T06:36:45.780711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785649+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:36:45.785649+0000","last_peered":"2024-04-22T06:36:45.785649+0000","last_clean":"2024-04-22T06:36:45.785649+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:36:45.785649+0000","last_undegraded":"2024-04-22T06:36:45.785649+0000","last_fullsized":"2024-04-22T06:36:45.785649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032177+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:36:46.032177+0000","last_peered":"2024-04-22T06:36:46.032177+0000","last_clean":"2024-04-22T06:36:46.032177+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:36:46.032177+0000","last_undegraded":"2024-04-22T06:36:46.032177+0000","last_fullsized":"2024-04-22T06:36:46.032177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:47.085069+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:36:47.085069+0000","last_peered":"2024-04-22T06:36:47.085069+0000","last_clean":"2024-04-22T06:36:47.085069+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:36:47.085069+0000","last_undegraded":"2024-04-22T06:36:47.085069+0000","last_fullsized":"2024-04-22T06:36:47.085069+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785500+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:36:45.785500+0000","last_peered":"2024-04-22T06:36:45.785500+0000","last_clean":"2024-04-22T06:36:45.785500+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:36:45.785500+0000","last_undegraded":"2024-04-22T06:36:45.785500+0000","last_fullsized":"2024-04-22T06:36:45.785500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785566+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:36:45.785566+0000","last_peered":"2024-04-22T06:36:45.785566+0000","last_clean":"2024-04-22T06:36:45.785566+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:36:45.785566+0000","last_undegraded":"2024-04-22T06:36:45.785566+0000","last_fullsized":"2024-04-22T06:36:45.785566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032187+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:36:46.032187+0000","last_peered":"2024-04-22T06:36:46.032187+0000","last_clean":"2024-04-22T06:36:46.032187+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:36:46.032187+0000","last_undegraded":"2024-04-22T06:36:46.032187+0000","last_fullsized":"2024-04-22T06:36:46.032187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782121+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:36:45.782121+0000","last_peered":"2024-04-22T06:36:45.782121+0000","last_clean":"2024-04-22T06:36:45.782121+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:36:45.782121+0000","last_undegraded":"2024-04-22T06:36:45.782121+0000","last_fullsized":"2024-04-22T06:36:45.782121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784870+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:36:45.784870+0000","last_peered":"2024-04-22T06:36:45.784870+0000","last_clean":"2024-04-22T06:36:45.784870+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:36:45.784870+0000","last_undegraded":"2024-04-22T06:36:45.784870+0000","last_fullsized":"2024-04-22T06:36:45.784870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031889+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:36:46.031889+0000","last_peered":"2024-04-22T06:36:46.031889+0000","last_clean":"2024-04-22T06:36:46.031889+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:36:46.031889+0000","last_undegraded":"2024-04-22T06:36:46.031889+0000","last_fullsized":"2024-04-22T06:36:46.031889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.779954+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:36:45.779954+0000","last_peered":"2024-04-22T06:36:45.779954+0000","last_clean":"2024-04-22T06:36:45.779954+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:36:45.779954+0000","last_undegraded":"2024-04-22T06:36:45.779954+0000","last_fullsized":"2024-04-22T06:36:45.779954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":155,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:51.089209+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:36:51.089209+0000","last_peered":"2024-04-22T06:36:51.089209+0000","last_clean":"2024-04-22T06:36:51.089209+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:36:51.089209+0000","last_undegraded":"2024-04-22T06:36:51.089209+0000","last_fullsized":"2024-04-22T06:36:51.089209+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780820+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:36:45.780820+0000","last_peered":"2024-04-22T06:36:45.780820+0000","last_clean":"2024-04-22T06:36:45.780820+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:36:45.780820+0000","last_undegraded":"2024-04-22T06:36:45.780820+0000","last_fullsized":"2024-04-22T06:36:45.780820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785609+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:36:45.785609+0000","last_peered":"2024-04-22T06:36:45.785609+0000","last_clean":"2024-04-22T06:36:45.785609+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:36:45.785609+0000","last_undegraded":"2024-04-22T06:36:45.785609+0000","last_fullsized":"2024-04-22T06:36:45.785609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032006+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:36:46.032006+0000","last_peered":"2024-04-22T06:36:46.032006+0000","last_clean":"2024-04-22T06:36:46.032006+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:36:46.032006+0000","last_undegraded":"2024-04-22T06:36:46.032006+0000","last_fullsized":"2024-04-22T06:36:46.032006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032315+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:36:46.032315+0000","last_peered":"2024-04-22T06:36:46.032315+0000","last_clean":"2024-04-22T06:36:46.032315+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:36:46.032315+0000","last_undegraded":"2024-04-22T06:36:46.032315+0000","last_fullsized":"2024-04-22T06:36:46.032315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.787055+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:36:45.787055+0000","last_peered":"2024-04-22T06:36:45.787055+0000","last_clean":"2024-04-22T06:36:45.787055+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:36:45.787055+0000","last_undegraded":"2024-04-22T06:36:45.787055+0000","last_fullsized":"2024-04-22T06:36:45.787055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":186,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:51.111524+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:36:51.111524+0000","last_peered":"2024-04-22T06:36:51.111524+0000","last_clean":"2024-04-22T06:36:51.111524+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:36:51.111524+0000","last_undegraded":"2024-04-22T06:36:51.111524+0000","last_fullsized":"2024-04-22T06:36:51.111524+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":127,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785509+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:36:45.785509+0000","last_peered":"2024-04-22T06:36:45.785509+0000","last_clean":"2024-04-22T06:36:45.785509+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:36:45.785509+0000","last_undegraded":"2024-04-22T06:36:45.785509+0000","last_fullsized":"2024-04-22T06:36:45.785509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.808639+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:36:45.808639+0000","last_peered":"2024-04-22T06:36:45.808639+0000","last_clean":"2024-04-22T06:36:45.808639+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:36:45.808639+0000","last_undegraded":"2024-04-22T06:36:45.808639+0000","last_fullsized":"2024-04-22T06:36:45.808639+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782201+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:36:45.782201+0000","last_peered":"2024-04-22T06:36:45.782201+0000","last_clean":"2024-04-22T06:36:45.782201+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:36:45.782201+0000","last_undegraded":"2024-04-22T06:36:45.782201+0000","last_fullsized":"2024-04-22T06:36:45.782201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032345+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:36:46.032345+0000","last_peered":"2024-04-22T06:36:46.032345+0000","last_clean":"2024-04-22T06:36:46.032345+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:36:46.032345+0000","last_undegraded":"2024-04-22T06:36:46.032345+0000","last_fullsized":"2024-04-22T06:36:46.032345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785328+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:36:45.785328+0000","last_peered":"2024-04-22T06:36:45.785328+0000","last_clean":"2024-04-22T06:36:45.785328+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:36:45.785328+0000","last_undegraded":"2024-04-22T06:36:45.785328+0000","last_fullsized":"2024-04-22T06:36:45.785328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:48.085675+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:36:48.085675+0000","last_peered":"2024-04-22T06:36:48.085675+0000","last_clean":"2024-04-22T06:36:48.085675+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:36:48.085675+0000","last_undegraded":"2024-04-22T06:36:48.085675+0000","last_fullsized":"2024-04-22T06:36:48.085675+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784747+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:36:45.784747+0000","last_peered":"2024-04-22T06:36:45.784747+0000","last_clean":"2024-04-22T06:36:45.784747+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:36:45.784747+0000","last_undegraded":"2024-04-22T06:36:45.784747+0000","last_fullsized":"2024-04-22T06:36:45.784747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785189+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:36:45.785189+0000","last_peered":"2024-04-22T06:36:45.785189+0000","last_clean":"2024-04-22T06:36:45.785189+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:36:45.785189+0000","last_undegraded":"2024-04-22T06:36:45.785189+0000","last_fullsized":"2024-04-22T06:36:45.785189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.033799+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:36:46.033799+0000","last_peered":"2024-04-22T06:36:46.033799+0000","last_clean":"2024-04-22T06:36:46.033799+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:36:46.033799+0000","last_undegraded":"2024-04-22T06:36:46.033799+0000","last_fullsized":"2024-04-22T06:36:46.033799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786021+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:36:45.786021+0000","last_peered":"2024-04-22T06:36:45.786021+0000","last_clean":"2024-04-22T06:36:45.786021+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:36:45.786021+0000","last_undegraded":"2024-04-22T06:36:45.786021+0000","last_fullsized":"2024-04-22T06:36:45.786021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782886+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:36:45.782886+0000","last_peered":"2024-04-22T06:36:45.782886+0000","last_clean":"2024-04-22T06:36:45.782886+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:36:45.782886+0000","last_undegraded":"2024-04-22T06:36:45.782886+0000","last_fullsized":"2024-04-22T06:36:45.782886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780147+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:36:45.780147+0000","last_peered":"2024-04-22T06:36:45.780147+0000","last_clean":"2024-04-22T06:36:45.780147+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:36:45.780147+0000","last_undegraded":"2024-04-22T06:36:45.780147+0000","last_fullsized":"2024-04-22T06:36:45.780147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786039+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:36:45.786039+0000","last_peered":"2024-04-22T06:36:45.786039+0000","last_clean":"2024-04-22T06:36:45.786039+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:36:45.786039+0000","last_undegraded":"2024-04-22T06:36:45.786039+0000","last_fullsized":"2024-04-22T06:36:45.786039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:47.084141+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:36:47.084141+0000","last_peered":"2024-04-22T06:36:47.084141+0000","last_clean":"2024-04-22T06:36:47.084141+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:36:47.084141+0000","last_undegraded":"2024-04-22T06:36:47.084141+0000","last_fullsized":"2024-04-22T06:36:47.084141+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785718+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:36:45.785718+0000","last_peered":"2024-04-22T06:36:45.785718+0000","last_clean":"2024-04-22T06:36:45.785718+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:36:45.785718+0000","last_undegraded":"2024-04-22T06:36:45.785718+0000","last_fullsized":"2024-04-22T06:36:45.785718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032168+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:36:46.032168+0000","last_peered":"2024-04-22T06:36:46.032168+0000","last_clean":"2024-04-22T06:36:46.032168+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:36:46.032168+0000","last_undegraded":"2024-04-22T06:36:46.032168+0000","last_fullsized":"2024-04-22T06:36:46.032168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:48.085208+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:36:48.085208+0000","last_peered":"2024-04-22T06:36:48.085208+0000","last_clean":"2024-04-22T06:36:48.085208+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:36:48.085208+0000","last_undegraded":"2024-04-22T06:36:48.085208+0000","last_fullsized":"2024-04-22T06:36:48.085208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":65,"num_read_kb":65,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785385+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:36:45.785385+0000","last_peered":"2024-04-22T06:36:45.785385+0000","last_clean":"2024-04-22T06:36:45.785385+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:36:45.785385+0000","last_undegraded":"2024-04-22T06:36:45.785385+0000","last_fullsized":"2024-04-22T06:36:45.785385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032289+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:36:46.032289+0000","last_peered":"2024-04-22T06:36:46.032289+0000","last_clean":"2024-04-22T06:36:46.032289+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:36:46.032289+0000","last_undegraded":"2024-04-22T06:36:46.032289+0000","last_fullsized":"2024-04-22T06:36:46.032289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784862+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:36:45.784862+0000","last_peered":"2024-04-22T06:36:45.784862+0000","last_clean":"2024-04-22T06:36:45.784862+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:36:45.784862+0000","last_undegraded":"2024-04-22T06:36:45.784862+0000","last_fullsized":"2024-04-22T06:36:45.784862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784112+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:36:45.784112+0000","last_peered":"2024-04-22T06:36:45.784112+0000","last_clean":"2024-04-22T06:36:45.784112+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:36:45.784112+0000","last_undegraded":"2024-04-22T06:36:45.784112+0000","last_fullsized":"2024-04-22T06:36:45.784112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785020+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:36:45.785020+0000","last_peered":"2024-04-22T06:36:45.785020+0000","last_clean":"2024-04-22T06:36:45.785020+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:36:45.785020+0000","last_undegraded":"2024-04-22T06:36:45.785020+0000","last_fullsized":"2024-04-22T06:36:45.785020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785630+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:36:45.785630+0000","last_peered":"2024-04-22T06:36:45.785630+0000","last_clean":"2024-04-22T06:36:45.785630+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:36:45.785630+0000","last_undegraded":"2024-04-22T06:36:45.785630+0000","last_fullsized":"2024-04-22T06:36:45.785630+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782240+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:36:45.782240+0000","last_peered":"2024-04-22T06:36:45.782240+0000","last_clean":"2024-04-22T06:36:45.782240+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:36:45.782240+0000","last_undegraded":"2024-04-22T06:36:45.782240+0000","last_fullsized":"2024-04-22T06:36:45.782240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780200+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:36:45.780200+0000","last_peered":"2024-04-22T06:36:45.780200+0000","last_clean":"2024-04-22T06:36:45.780200+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:36:45.780200+0000","last_undegraded":"2024-04-22T06:36:45.780200+0000","last_fullsized":"2024-04-22T06:36:45.780200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785733+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:36:45.785733+0000","last_peered":"2024-04-22T06:36:45.785733+0000","last_clean":"2024-04-22T06:36:45.785733+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:36:45.785733+0000","last_undegraded":"2024-04-22T06:36:45.785733+0000","last_fullsized":"2024-04-22T06:36:45.785733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032162+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:36:46.032162+0000","last_peered":"2024-04-22T06:36:46.032162+0000","last_clean":"2024-04-22T06:36:46.032162+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:36:46.032162+0000","last_undegraded":"2024-04-22T06:36:46.032162+0000","last_fullsized":"2024-04-22T06:36:46.032162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031982+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:36:46.031982+0000","last_peered":"2024-04-22T06:36:46.031982+0000","last_clean":"2024-04-22T06:36:46.031982+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:36:46.031982+0000","last_undegraded":"2024-04-22T06:36:46.031982+0000","last_fullsized":"2024-04-22T06:36:46.031982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780540+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:36:45.780540+0000","last_peered":"2024-04-22T06:36:45.780540+0000","last_clean":"2024-04-22T06:36:45.780540+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:36:45.780540+0000","last_undegraded":"2024-04-22T06:36:45.780540+0000","last_fullsized":"2024-04-22T06:36:45.780540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785081+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:36:45.785081+0000","last_peered":"2024-04-22T06:36:45.785081+0000","last_clean":"2024-04-22T06:36:45.785081+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:36:45.785081+0000","last_undegraded":"2024-04-22T06:36:45.785081+0000","last_fullsized":"2024-04-22T06:36:45.785081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784570+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:36:45.784570+0000","last_peered":"2024-04-22T06:36:45.784570+0000","last_clean":"2024-04-22T06:36:45.784570+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:36:45.784570+0000","last_undegraded":"2024-04-22T06:36:45.784570+0000","last_fullsized":"2024-04-22T06:36:45.784570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782399+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:36:45.782399+0000","last_peered":"2024-04-22T06:36:45.782399+0000","last_clean":"2024-04-22T06:36:45.782399+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:36:45.782399+0000","last_undegraded":"2024-04-22T06:36:45.782399+0000","last_fullsized":"2024-04-22T06:36:45.782399+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780266+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:36:45.780266+0000","last_peered":"2024-04-22T06:36:45.780266+0000","last_clean":"2024-04-22T06:36:45.780266+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:36:45.780266+0000","last_undegraded":"2024-04-22T06:36:45.780266+0000","last_fullsized":"2024-04-22T06:36:45.780266+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785241+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:36:45.785241+0000","last_peered":"2024-04-22T06:36:45.785241+0000","last_clean":"2024-04-22T06:36:45.785241+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:36:45.785241+0000","last_undegraded":"2024-04-22T06:36:45.785241+0000","last_fullsized":"2024-04-22T06:36:45.785241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785647+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:36:45.785647+0000","last_peered":"2024-04-22T06:36:45.785647+0000","last_clean":"2024-04-22T06:36:45.785647+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:36:45.785647+0000","last_undegraded":"2024-04-22T06:36:45.785647+0000","last_fullsized":"2024-04-22T06:36:45.785647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785703+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:36:45.785703+0000","last_peered":"2024-04-22T06:36:45.785703+0000","last_clean":"2024-04-22T06:36:45.785703+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:36:45.785703+0000","last_undegraded":"2024-04-22T06:36:45.785703+0000","last_fullsized":"2024-04-22T06:36:45.785703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:49.087728+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:36:49.087728+0000","last_peered":"2024-04-22T06:36:49.087728+0000","last_clean":"2024-04-22T06:36:49.087728+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:36:49.087728+0000","last_undegraded":"2024-04-22T06:36:49.087728+0000","last_fullsized":"2024-04-22T06:36:49.087728+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785177+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:36:45.785177+0000","last_peered":"2024-04-22T06:36:45.785177+0000","last_clean":"2024-04-22T06:36:45.785177+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:36:45.785177+0000","last_undegraded":"2024-04-22T06:36:45.785177+0000","last_fullsized":"2024-04-22T06:36:45.785177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032174+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:36:46.032174+0000","last_peered":"2024-04-22T06:36:46.032174+0000","last_clean":"2024-04-22T06:36:46.032174+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:36:46.032174+0000","last_undegraded":"2024-04-22T06:36:46.032174+0000","last_fullsized":"2024-04-22T06:36:46.032174+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785473+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:36:45.785473+0000","last_peered":"2024-04-22T06:36:45.785473+0000","last_clean":"2024-04-22T06:36:45.785473+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:36:45.785473+0000","last_undegraded":"2024-04-22T06:36:45.785473+0000","last_fullsized":"2024-04-22T06:36:45.785473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782251+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:36:45.782251+0000","last_peered":"2024-04-22T06:36:45.782251+0000","last_clean":"2024-04-22T06:36:45.782251+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:36:45.782251+0000","last_undegraded":"2024-04-22T06:36:45.782251+0000","last_fullsized":"2024-04-22T06:36:45.782251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785814+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:36:45.785814+0000","last_peered":"2024-04-22T06:36:45.785814+0000","last_clean":"2024-04-22T06:36:45.785814+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:36:45.785814+0000","last_undegraded":"2024-04-22T06:36:45.785814+0000","last_fullsized":"2024-04-22T06:36:45.785814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782532+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:36:45.782532+0000","last_peered":"2024-04-22T06:36:45.782532+0000","last_clean":"2024-04-22T06:36:45.782532+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:36:45.782532+0000","last_undegraded":"2024-04-22T06:36:45.782532+0000","last_fullsized":"2024-04-22T06:36:45.782532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780142+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:36:45.780142+0000","last_peered":"2024-04-22T06:36:45.780142+0000","last_clean":"2024-04-22T06:36:45.780142+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:36:45.780142+0000","last_undegraded":"2024-04-22T06:36:45.780142+0000","last_fullsized":"2024-04-22T06:36:45.780142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780125+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:36:45.780125+0000","last_peered":"2024-04-22T06:36:45.780125+0000","last_clean":"2024-04-22T06:36:45.780125+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:36:45.780125+0000","last_undegraded":"2024-04-22T06:36:45.780125+0000","last_fullsized":"2024-04-22T06:36:45.780125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031857+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:36:46.031857+0000","last_peered":"2024-04-22T06:36:46.031857+0000","last_clean":"2024-04-22T06:36:46.031857+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:36:46.031857+0000","last_undegraded":"2024-04-22T06:36:46.031857+0000","last_fullsized":"2024-04-22T06:36:46.031857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782370+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:36:45.782370+0000","last_peered":"2024-04-22T06:36:45.782370+0000","last_clean":"2024-04-22T06:36:45.782370+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:36:45.782370+0000","last_undegraded":"2024-04-22T06:36:45.782370+0000","last_fullsized":"2024-04-22T06:36:45.782370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783162+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:36:45.783162+0000","last_peered":"2024-04-22T06:36:45.783162+0000","last_clean":"2024-04-22T06:36:45.783162+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:36:45.783162+0000","last_undegraded":"2024-04-22T06:36:45.783162+0000","last_fullsized":"2024-04-22T06:36:45.783162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780706+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:36:45.780706+0000","last_peered":"2024-04-22T06:36:45.780706+0000","last_clean":"2024-04-22T06:36:45.780706+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:36:45.780706+0000","last_undegraded":"2024-04-22T06:36:45.780706+0000","last_fullsized":"2024-04-22T06:36:45.780706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785720+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:36:45.785720+0000","last_peered":"2024-04-22T06:36:45.785720+0000","last_clean":"2024-04-22T06:36:45.785720+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:36:45.785720+0000","last_undegraded":"2024-04-22T06:36:45.785720+0000","last_fullsized":"2024-04-22T06:36:45.785720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031911+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:36:46.031911+0000","last_peered":"2024-04-22T06:36:46.031911+0000","last_clean":"2024-04-22T06:36:46.031911+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:36:46.031911+0000","last_undegraded":"2024-04-22T06:36:46.031911+0000","last_fullsized":"2024-04-22T06:36:46.031911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784412+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:36:45.784412+0000","last_peered":"2024-04-22T06:36:45.784412+0000","last_clean":"2024-04-22T06:36:45.784412+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:36:45.784412+0000","last_undegraded":"2024-04-22T06:36:45.784412+0000","last_fullsized":"2024-04-22T06:36:45.784412+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780917+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:36:45.780917+0000","last_peered":"2024-04-22T06:36:45.780917+0000","last_clean":"2024-04-22T06:36:45.780917+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:36:45.780917+0000","last_undegraded":"2024-04-22T06:36:45.780917+0000","last_fullsized":"2024-04-22T06:36:45.780917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782398+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:36:45.782398+0000","last_peered":"2024-04-22T06:36:45.782398+0000","last_clean":"2024-04-22T06:36:45.782398+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:36:45.782398+0000","last_undegraded":"2024-04-22T06:36:45.782398+0000","last_fullsized":"2024-04-22T06:36:45.782398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782642+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:36:45.782642+0000","last_peered":"2024-04-22T06:36:45.782642+0000","last_clean":"2024-04-22T06:36:45.782642+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:36:45.782642+0000","last_undegraded":"2024-04-22T06:36:45.782642+0000","last_fullsized":"2024-04-22T06:36:45.782642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66278,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":680,"num_read_kb":415,"num_write":235,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":476683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514529,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1695744,"data_stored":1127500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710758,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":476545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972395,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":476923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266735,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1613824,"data_stored":1069441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430261,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":476262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1024000,"data_stored":477601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":536792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":476087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":476514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691896,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32772,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712380,"statfs":{"total":95995035648,"available":95961477120,"internally_reserved":0,"allocated":1626112,"data_stored":1068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:36:55.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T06:36:55.287 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:55.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:55.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:55 smithi105 ceph-mon[27916]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:55.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:55 smithi112 ceph-mon[31236]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s; 6 B/s, 2 keys/s, 1 objects/s recovering 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:36:55.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:55 smithi177 ceph-mon[31346]: from='mgr.14152 172.21.15.105:0/3984762534' entity='mgr.x' 2024-04-22T06:36:56.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 -- 172.21.15.105:0/3904931956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41019d0 msgr2=0x7fc8d4101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/3904931956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41019d0 0x7fc8d4101e50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009990 tx=0x7fc8bc009ca0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 -- 172.21.15.105:0/3904931956 shutdown_connections 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/3904931956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d4102390 0x7fc8d410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/3904931956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/3904931956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d41019d0 0x7fc8d4101e50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 -- 172.21.15.105:0/3904931956 >> 172.21.15.105:0/3904931956 conn(0x7fc8d4077990 msgr2=0x7fc8d4077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 -- 172.21.15.105:0/3904931956 shutdown_connections 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 -- 172.21.15.105:0/3904931956 wait complete. 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.822+0000 7fc8d99b1700 1 Processor -- start 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 -- start start 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d41019d0 0x7fc8d4199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:56.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d4111b50 con 0x7fc8d4102390 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d41119d0 con 0x7fc8d41019d0 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d99b1700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d4111cd0 con 0x7fc8d4107940 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52824/0 (socket says 172.21.15.105:52824) 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 -- 172.21.15.105:0/4042244177 learned_addr learned my addr 172.21.15.105:0/4042244177 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d37fe700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d2ffd700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d41019d0 0x7fc8d4199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 -- 172.21.15.105:0/4042244177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 msgr2=0x7fc8d419d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d419d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 -- 172.21.15.105:0/4042244177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d41019d0 msgr2=0x7fc8d4199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d41019d0 0x7fc8d4199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 -- 172.21.15.105:0/4042244177 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d419e110 con 0x7fc8d4102390 2024-04-22T06:36:56.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8d37fe700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d419d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.823+0000 7fc8ca7fc700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc000c00 tx=0x7fc8bc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.824+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc027070 con 0x7fc8d4102390 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.824+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d419e3a0 con 0x7fc8d4102390 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.824+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8bc022470 con 0x7fc8d4102390 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.824+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc017650 con 0x7fc8d4102390 2024-04-22T06:36:56.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.824+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d41a5c80 con 0x7fc8d4102390 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.825+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d4069730 con 0x7fc8d4102390 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.826+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc8bc02a070 con 0x7fc8d4102390 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.826+0000 7fc8d89af700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 0x7fc8c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.826+0000 7fc8d2ffd700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 0x7fc8c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.827+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc8bc0d17b0 con 0x7fc8d4102390 2024-04-22T06:36:56.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.827+0000 7fc8d2ffd700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 0x7fc8c00a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009bd0 tx=0x7fc8c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:56.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:56.830+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8bc013070 con 0x7fc8d4102390 2024-04-22T06:36:57.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.025+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8d419a460 con 0x7fc8c00a3cd0 2024-04-22T06:36:57.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.032+0000 7fc8d89af700 1 -- 172.21.15.105:0/4042244177 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258776 (secure 0 0 0) 0x7fc8d419a460 con 0x7fc8c00a3cd0 2024-04-22T06:36:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:36:57.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 msgr2=0x7fc8c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:57.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 0x7fc8c00a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009bd0 tx=0x7fc8c4009260 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 msgr2=0x7fc8d4199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc000c00 tx=0x7fc8bc009580 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 shutdown_connections 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc8c00a3cd0 0x7fc8c00a6190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d41019d0 0x7fc8d4199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d4107940 0x7fc8d419d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 --2- 172.21.15.105:0/4042244177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d4102390 0x7fc8d4199640 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 >> 172.21.15.105:0/4042244177 conn(0x7fc8d4077990 msgr2=0x7fc8d406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 shutdown_connections 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:57.035+0000 7fc8d99b1700 1 -- 172.21.15.105:0/4042244177 wait complete. 2024-04-22T06:36:57.037 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:36:57.742 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-04-22T06:36:56.638220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656880,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":845,"num_read_kb":544,"num_write":418,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375536,"kb_used_data":13836,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566288,"statfs":{"total":1151940427776,"available":1151555878912,"internally_reserved":0,"allocated":14168064,"data_stored":7675716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":99,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.859495"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783094+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:36:45.783094+0000","last_peered":"2024-04-22T06:36:45.783094+0000","last_clean":"2024-04-22T06:36:45.783094+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:36:45.783094+0000","last_undegraded":"2024-04-22T06:36:45.783094+0000","last_fullsized":"2024-04-22T06:36:45.783094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032004+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:36:46.032004+0000","last_peered":"2024-04-22T06:36:46.032004+0000","last_clean":"2024-04-22T06:36:46.032004+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:36:46.032004+0000","last_undegraded":"2024-04-22T06:36:46.032004+0000","last_fullsized":"2024-04-22T06:36:46.032004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031923+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:36:46.031923+0000","last_peered":"2024-04-22T06:36:46.031923+0000","last_clean":"2024-04-22T06:36:46.031923+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:36:46.031923+0000","last_undegraded":"2024-04-22T06:36:46.031923+0000","last_fullsized":"2024-04-22T06:36:46.031923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032350+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:36:46.032350+0000","last_peered":"2024-04-22T06:36:46.032350+0000","last_clean":"2024-04-22T06:36:46.032350+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:36:46.032350+0000","last_undegraded":"2024-04-22T06:36:46.032350+0000","last_fullsized":"2024-04-22T06:36:46.032350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785549+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:36:45.785549+0000","last_peered":"2024-04-22T06:36:45.785549+0000","last_clean":"2024-04-22T06:36:45.785549+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:36:45.785549+0000","last_undegraded":"2024-04-22T06:36:45.785549+0000","last_fullsized":"2024-04-22T06:36:45.785549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785686+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:36:45.785686+0000","last_peered":"2024-04-22T06:36:45.785686+0000","last_clean":"2024-04-22T06:36:45.785686+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:36:45.785686+0000","last_undegraded":"2024-04-22T06:36:45.785686+0000","last_fullsized":"2024-04-22T06:36:45.785686+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785109+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:36:45.785109+0000","last_peered":"2024-04-22T06:36:45.785109+0000","last_clean":"2024-04-22T06:36:45.785109+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:36:45.785109+0000","last_undegraded":"2024-04-22T06:36:45.785109+0000","last_fullsized":"2024-04-22T06:36:45.785109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784799+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:36:45.784799+0000","last_peered":"2024-04-22T06:36:45.784799+0000","last_clean":"2024-04-22T06:36:45.784799+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:36:45.784799+0000","last_undegraded":"2024-04-22T06:36:45.784799+0000","last_fullsized":"2024-04-22T06:36:45.784799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.781064+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:36:45.781064+0000","last_peered":"2024-04-22T06:36:45.781064+0000","last_clean":"2024-04-22T06:36:45.781064+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:36:45.781064+0000","last_undegraded":"2024-04-22T06:36:45.781064+0000","last_fullsized":"2024-04-22T06:36:45.781064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786201+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:36:45.786201+0000","last_peered":"2024-04-22T06:36:45.786201+0000","last_clean":"2024-04-22T06:36:45.786201+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:36:45.786201+0000","last_undegraded":"2024-04-22T06:36:45.786201+0000","last_fullsized":"2024-04-22T06:36:45.786201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785024+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:36:45.785024+0000","last_peered":"2024-04-22T06:36:45.785024+0000","last_clean":"2024-04-22T06:36:45.785024+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:36:45.785024+0000","last_undegraded":"2024-04-22T06:36:45.785024+0000","last_fullsized":"2024-04-22T06:36:45.785024+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.788195+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:36:45.788195+0000","last_peered":"2024-04-22T06:36:45.788195+0000","last_clean":"2024-04-22T06:36:45.788195+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:36:45.788195+0000","last_undegraded":"2024-04-22T06:36:45.788195+0000","last_fullsized":"2024-04-22T06:36:45.788195+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782412+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:36:45.782412+0000","last_peered":"2024-04-22T06:36:45.782412+0000","last_clean":"2024-04-22T06:36:45.782412+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:36:45.782412+0000","last_undegraded":"2024-04-22T06:36:45.782412+0000","last_fullsized":"2024-04-22T06:36:45.782412+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031721+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:36:46.031721+0000","last_peered":"2024-04-22T06:36:46.031721+0000","last_clean":"2024-04-22T06:36:46.031721+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:36:46.031721+0000","last_undegraded":"2024-04-22T06:36:46.031721+0000","last_fullsized":"2024-04-22T06:36:46.031721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031655+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:36:46.031655+0000","last_peered":"2024-04-22T06:36:46.031655+0000","last_clean":"2024-04-22T06:36:46.031655+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:36:46.031655+0000","last_undegraded":"2024-04-22T06:36:46.031655+0000","last_fullsized":"2024-04-22T06:36:46.031655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780414+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:36:45.780414+0000","last_peered":"2024-04-22T06:36:45.780414+0000","last_clean":"2024-04-22T06:36:45.780414+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:36:45.780414+0000","last_undegraded":"2024-04-22T06:36:45.780414+0000","last_fullsized":"2024-04-22T06:36:45.780414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783262+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:36:45.783262+0000","last_peered":"2024-04-22T06:36:45.783262+0000","last_clean":"2024-04-22T06:36:45.783262+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:36:45.783262+0000","last_undegraded":"2024-04-22T06:36:45.783262+0000","last_fullsized":"2024-04-22T06:36:45.783262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780172+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:36:45.780172+0000","last_peered":"2024-04-22T06:36:45.780172+0000","last_clean":"2024-04-22T06:36:45.780172+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:36:45.780172+0000","last_undegraded":"2024-04-22T06:36:45.780172+0000","last_fullsized":"2024-04-22T06:36:45.780172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780471+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:36:45.780471+0000","last_peered":"2024-04-22T06:36:45.780471+0000","last_clean":"2024-04-22T06:36:45.780471+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:36:45.780471+0000","last_undegraded":"2024-04-22T06:36:45.780471+0000","last_fullsized":"2024-04-22T06:36:45.780471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:52.089692+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:36:52.089692+0000","last_peered":"2024-04-22T06:36:52.089692+0000","last_clean":"2024-04-22T06:36:52.089692+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:36:52.089692+0000","last_undegraded":"2024-04-22T06:36:52.089692+0000","last_fullsized":"2024-04-22T06:36:52.089692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785601+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:36:45.785601+0000","last_peered":"2024-04-22T06:36:45.785601+0000","last_clean":"2024-04-22T06:36:45.785601+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:36:45.785601+0000","last_undegraded":"2024-04-22T06:36:45.785601+0000","last_fullsized":"2024-04-22T06:36:45.785601+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785737+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:36:45.785737+0000","last_peered":"2024-04-22T06:36:45.785737+0000","last_clean":"2024-04-22T06:36:45.785737+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:36:45.785737+0000","last_undegraded":"2024-04-22T06:36:45.785737+0000","last_fullsized":"2024-04-22T06:36:45.785737+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032036+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:36:46.032036+0000","last_peered":"2024-04-22T06:36:46.032036+0000","last_clean":"2024-04-22T06:36:46.032036+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:36:46.032036+0000","last_undegraded":"2024-04-22T06:36:46.032036+0000","last_fullsized":"2024-04-22T06:36:46.032036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.799058+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:36:45.799058+0000","last_peered":"2024-04-22T06:36:45.799058+0000","last_clean":"2024-04-22T06:36:45.799058+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:36:45.799058+0000","last_undegraded":"2024-04-22T06:36:45.799058+0000","last_fullsized":"2024-04-22T06:36:45.799058+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780711+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:36:45.780711+0000","last_peered":"2024-04-22T06:36:45.780711+0000","last_clean":"2024-04-22T06:36:45.780711+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:36:45.780711+0000","last_undegraded":"2024-04-22T06:36:45.780711+0000","last_fullsized":"2024-04-22T06:36:45.780711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785649+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:36:45.785649+0000","last_peered":"2024-04-22T06:36:45.785649+0000","last_clean":"2024-04-22T06:36:45.785649+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:36:45.785649+0000","last_undegraded":"2024-04-22T06:36:45.785649+0000","last_fullsized":"2024-04-22T06:36:45.785649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032177+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:36:46.032177+0000","last_peered":"2024-04-22T06:36:46.032177+0000","last_clean":"2024-04-22T06:36:46.032177+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:36:46.032177+0000","last_undegraded":"2024-04-22T06:36:46.032177+0000","last_fullsized":"2024-04-22T06:36:46.032177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:52.091287+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:36:52.091287+0000","last_peered":"2024-04-22T06:36:52.091287+0000","last_clean":"2024-04-22T06:36:52.091287+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:36:52.091287+0000","last_undegraded":"2024-04-22T06:36:52.091287+0000","last_fullsized":"2024-04-22T06:36:52.091287+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785500+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:36:45.785500+0000","last_peered":"2024-04-22T06:36:45.785500+0000","last_clean":"2024-04-22T06:36:45.785500+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:36:45.785500+0000","last_undegraded":"2024-04-22T06:36:45.785500+0000","last_fullsized":"2024-04-22T06:36:45.785500+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785566+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:36:45.785566+0000","last_peered":"2024-04-22T06:36:45.785566+0000","last_clean":"2024-04-22T06:36:45.785566+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:36:45.785566+0000","last_undegraded":"2024-04-22T06:36:45.785566+0000","last_fullsized":"2024-04-22T06:36:45.785566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032187+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:36:46.032187+0000","last_peered":"2024-04-22T06:36:46.032187+0000","last_clean":"2024-04-22T06:36:46.032187+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:36:46.032187+0000","last_undegraded":"2024-04-22T06:36:46.032187+0000","last_fullsized":"2024-04-22T06:36:46.032187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782121+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:36:45.782121+0000","last_peered":"2024-04-22T06:36:45.782121+0000","last_clean":"2024-04-22T06:36:45.782121+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:36:45.782121+0000","last_undegraded":"2024-04-22T06:36:45.782121+0000","last_fullsized":"2024-04-22T06:36:45.782121+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784870+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:36:45.784870+0000","last_peered":"2024-04-22T06:36:45.784870+0000","last_clean":"2024-04-22T06:36:45.784870+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:36:45.784870+0000","last_undegraded":"2024-04-22T06:36:45.784870+0000","last_fullsized":"2024-04-22T06:36:45.784870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031889+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:36:46.031889+0000","last_peered":"2024-04-22T06:36:46.031889+0000","last_clean":"2024-04-22T06:36:46.031889+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:36:46.031889+0000","last_undegraded":"2024-04-22T06:36:46.031889+0000","last_fullsized":"2024-04-22T06:36:46.031889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.779954+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:36:45.779954+0000","last_peered":"2024-04-22T06:36:45.779954+0000","last_clean":"2024-04-22T06:36:45.779954+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:36:45.779954+0000","last_undegraded":"2024-04-22T06:36:45.779954+0000","last_fullsized":"2024-04-22T06:36:45.779954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":155,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:51.089209+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:36:51.089209+0000","last_peered":"2024-04-22T06:36:51.089209+0000","last_clean":"2024-04-22T06:36:51.089209+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:36:51.089209+0000","last_undegraded":"2024-04-22T06:36:51.089209+0000","last_fullsized":"2024-04-22T06:36:51.089209+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780820+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:36:45.780820+0000","last_peered":"2024-04-22T06:36:45.780820+0000","last_clean":"2024-04-22T06:36:45.780820+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:36:45.780820+0000","last_undegraded":"2024-04-22T06:36:45.780820+0000","last_fullsized":"2024-04-22T06:36:45.780820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785609+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:36:45.785609+0000","last_peered":"2024-04-22T06:36:45.785609+0000","last_clean":"2024-04-22T06:36:45.785609+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:36:45.785609+0000","last_undegraded":"2024-04-22T06:36:45.785609+0000","last_fullsized":"2024-04-22T06:36:45.785609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032006+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:36:46.032006+0000","last_peered":"2024-04-22T06:36:46.032006+0000","last_clean":"2024-04-22T06:36:46.032006+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:36:46.032006+0000","last_undegraded":"2024-04-22T06:36:46.032006+0000","last_fullsized":"2024-04-22T06:36:46.032006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032315+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:36:46.032315+0000","last_peered":"2024-04-22T06:36:46.032315+0000","last_clean":"2024-04-22T06:36:46.032315+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:36:46.032315+0000","last_undegraded":"2024-04-22T06:36:46.032315+0000","last_fullsized":"2024-04-22T06:36:46.032315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.787055+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:36:45.787055+0000","last_peered":"2024-04-22T06:36:45.787055+0000","last_clean":"2024-04-22T06:36:45.787055+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:36:45.787055+0000","last_undegraded":"2024-04-22T06:36:45.787055+0000","last_fullsized":"2024-04-22T06:36:45.787055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'29","reported_seq":186,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:51.111524+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:36:51.111524+0000","last_peered":"2024-04-22T06:36:51.111524+0000","last_clean":"2024-04-22T06:36:51.111524+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:36:51.111524+0000","last_undegraded":"2024-04-22T06:36:51.111524+0000","last_fullsized":"2024-04-22T06:36:51.111524+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":127,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785509+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:36:45.785509+0000","last_peered":"2024-04-22T06:36:45.785509+0000","last_clean":"2024-04-22T06:36:45.785509+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:36:45.785509+0000","last_undegraded":"2024-04-22T06:36:45.785509+0000","last_fullsized":"2024-04-22T06:36:45.785509+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.808639+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:36:45.808639+0000","last_peered":"2024-04-22T06:36:45.808639+0000","last_clean":"2024-04-22T06:36:45.808639+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:36:45.808639+0000","last_undegraded":"2024-04-22T06:36:45.808639+0000","last_fullsized":"2024-04-22T06:36:45.808639+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782201+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:36:45.782201+0000","last_peered":"2024-04-22T06:36:45.782201+0000","last_clean":"2024-04-22T06:36:45.782201+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:36:45.782201+0000","last_undegraded":"2024-04-22T06:36:45.782201+0000","last_fullsized":"2024-04-22T06:36:45.782201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032345+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:36:46.032345+0000","last_peered":"2024-04-22T06:36:46.032345+0000","last_clean":"2024-04-22T06:36:46.032345+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:36:46.032345+0000","last_undegraded":"2024-04-22T06:36:46.032345+0000","last_fullsized":"2024-04-22T06:36:46.032345+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785328+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:36:45.785328+0000","last_peered":"2024-04-22T06:36:45.785328+0000","last_clean":"2024-04-22T06:36:45.785328+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:36:45.785328+0000","last_undegraded":"2024-04-22T06:36:45.785328+0000","last_fullsized":"2024-04-22T06:36:45.785328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":96,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:53.091896+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:36:53.091896+0000","last_peered":"2024-04-22T06:36:53.091896+0000","last_clean":"2024-04-22T06:36:53.091896+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:36:53.091896+0000","last_undegraded":"2024-04-22T06:36:53.091896+0000","last_fullsized":"2024-04-22T06:36:53.091896+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784747+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:36:45.784747+0000","last_peered":"2024-04-22T06:36:45.784747+0000","last_clean":"2024-04-22T06:36:45.784747+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:36:45.784747+0000","last_undegraded":"2024-04-22T06:36:45.784747+0000","last_fullsized":"2024-04-22T06:36:45.784747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785189+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:36:45.785189+0000","last_peered":"2024-04-22T06:36:45.785189+0000","last_clean":"2024-04-22T06:36:45.785189+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:36:45.785189+0000","last_undegraded":"2024-04-22T06:36:45.785189+0000","last_fullsized":"2024-04-22T06:36:45.785189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.033799+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:36:46.033799+0000","last_peered":"2024-04-22T06:36:46.033799+0000","last_clean":"2024-04-22T06:36:46.033799+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:36:46.033799+0000","last_undegraded":"2024-04-22T06:36:46.033799+0000","last_fullsized":"2024-04-22T06:36:46.033799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786021+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:36:45.786021+0000","last_peered":"2024-04-22T06:36:45.786021+0000","last_clean":"2024-04-22T06:36:45.786021+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:36:45.786021+0000","last_undegraded":"2024-04-22T06:36:45.786021+0000","last_fullsized":"2024-04-22T06:36:45.786021+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782886+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:36:45.782886+0000","last_peered":"2024-04-22T06:36:45.782886+0000","last_clean":"2024-04-22T06:36:45.782886+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:36:45.782886+0000","last_undegraded":"2024-04-22T06:36:45.782886+0000","last_fullsized":"2024-04-22T06:36:45.782886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780147+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:36:45.780147+0000","last_peered":"2024-04-22T06:36:45.780147+0000","last_clean":"2024-04-22T06:36:45.780147+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:36:45.780147+0000","last_undegraded":"2024-04-22T06:36:45.780147+0000","last_fullsized":"2024-04-22T06:36:45.780147+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.786039+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:36:45.786039+0000","last_peered":"2024-04-22T06:36:45.786039+0000","last_clean":"2024-04-22T06:36:45.786039+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:36:45.786039+0000","last_undegraded":"2024-04-22T06:36:45.786039+0000","last_fullsized":"2024-04-22T06:36:45.786039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:52.090201+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:36:52.090201+0000","last_peered":"2024-04-22T06:36:52.090201+0000","last_clean":"2024-04-22T06:36:52.090201+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:36:52.090201+0000","last_undegraded":"2024-04-22T06:36:52.090201+0000","last_fullsized":"2024-04-22T06:36:52.090201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785718+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:36:45.785718+0000","last_peered":"2024-04-22T06:36:45.785718+0000","last_clean":"2024-04-22T06:36:45.785718+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:36:45.785718+0000","last_undegraded":"2024-04-22T06:36:45.785718+0000","last_fullsized":"2024-04-22T06:36:45.785718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032168+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:36:46.032168+0000","last_peered":"2024-04-22T06:36:46.032168+0000","last_clean":"2024-04-22T06:36:46.032168+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:36:46.032168+0000","last_undegraded":"2024-04-22T06:36:46.032168+0000","last_fullsized":"2024-04-22T06:36:46.032168+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:53.091343+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:36:53.091343+0000","last_peered":"2024-04-22T06:36:53.091343+0000","last_clean":"2024-04-22T06:36:53.091343+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:36:53.091343+0000","last_undegraded":"2024-04-22T06:36:53.091343+0000","last_fullsized":"2024-04-22T06:36:53.091343+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785385+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:36:45.785385+0000","last_peered":"2024-04-22T06:36:45.785385+0000","last_clean":"2024-04-22T06:36:45.785385+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:36:45.785385+0000","last_undegraded":"2024-04-22T06:36:45.785385+0000","last_fullsized":"2024-04-22T06:36:45.785385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032289+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:36:46.032289+0000","last_peered":"2024-04-22T06:36:46.032289+0000","last_clean":"2024-04-22T06:36:46.032289+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:36:46.032289+0000","last_undegraded":"2024-04-22T06:36:46.032289+0000","last_fullsized":"2024-04-22T06:36:46.032289+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784862+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:36:45.784862+0000","last_peered":"2024-04-22T06:36:45.784862+0000","last_clean":"2024-04-22T06:36:45.784862+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:36:45.784862+0000","last_undegraded":"2024-04-22T06:36:45.784862+0000","last_fullsized":"2024-04-22T06:36:45.784862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784112+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:36:45.784112+0000","last_peered":"2024-04-22T06:36:45.784112+0000","last_clean":"2024-04-22T06:36:45.784112+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:36:45.784112+0000","last_undegraded":"2024-04-22T06:36:45.784112+0000","last_fullsized":"2024-04-22T06:36:45.784112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785020+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:36:45.785020+0000","last_peered":"2024-04-22T06:36:45.785020+0000","last_clean":"2024-04-22T06:36:45.785020+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:36:45.785020+0000","last_undegraded":"2024-04-22T06:36:45.785020+0000","last_fullsized":"2024-04-22T06:36:45.785020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785630+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:36:45.785630+0000","last_peered":"2024-04-22T06:36:45.785630+0000","last_clean":"2024-04-22T06:36:45.785630+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:36:45.785630+0000","last_undegraded":"2024-04-22T06:36:45.785630+0000","last_fullsized":"2024-04-22T06:36:45.785630+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782240+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:36:45.782240+0000","last_peered":"2024-04-22T06:36:45.782240+0000","last_clean":"2024-04-22T06:36:45.782240+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:36:45.782240+0000","last_undegraded":"2024-04-22T06:36:45.782240+0000","last_fullsized":"2024-04-22T06:36:45.782240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780200+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:36:45.780200+0000","last_peered":"2024-04-22T06:36:45.780200+0000","last_clean":"2024-04-22T06:36:45.780200+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:36:45.780200+0000","last_undegraded":"2024-04-22T06:36:45.780200+0000","last_fullsized":"2024-04-22T06:36:45.780200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785733+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:36:45.785733+0000","last_peered":"2024-04-22T06:36:45.785733+0000","last_clean":"2024-04-22T06:36:45.785733+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:36:45.785733+0000","last_undegraded":"2024-04-22T06:36:45.785733+0000","last_fullsized":"2024-04-22T06:36:45.785733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032162+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:36:46.032162+0000","last_peered":"2024-04-22T06:36:46.032162+0000","last_clean":"2024-04-22T06:36:46.032162+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:36:46.032162+0000","last_undegraded":"2024-04-22T06:36:46.032162+0000","last_fullsized":"2024-04-22T06:36:46.032162+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031982+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:36:46.031982+0000","last_peered":"2024-04-22T06:36:46.031982+0000","last_clean":"2024-04-22T06:36:46.031982+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:36:46.031982+0000","last_undegraded":"2024-04-22T06:36:46.031982+0000","last_fullsized":"2024-04-22T06:36:46.031982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780540+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:36:45.780540+0000","last_peered":"2024-04-22T06:36:45.780540+0000","last_clean":"2024-04-22T06:36:45.780540+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:36:45.780540+0000","last_undegraded":"2024-04-22T06:36:45.780540+0000","last_fullsized":"2024-04-22T06:36:45.780540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785081+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:36:45.785081+0000","last_peered":"2024-04-22T06:36:45.785081+0000","last_clean":"2024-04-22T06:36:45.785081+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:36:45.785081+0000","last_undegraded":"2024-04-22T06:36:45.785081+0000","last_fullsized":"2024-04-22T06:36:45.785081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784570+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:36:45.784570+0000","last_peered":"2024-04-22T06:36:45.784570+0000","last_clean":"2024-04-22T06:36:45.784570+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:36:45.784570+0000","last_undegraded":"2024-04-22T06:36:45.784570+0000","last_fullsized":"2024-04-22T06:36:45.784570+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782399+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:36:45.782399+0000","last_peered":"2024-04-22T06:36:45.782399+0000","last_clean":"2024-04-22T06:36:45.782399+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:36:45.782399+0000","last_undegraded":"2024-04-22T06:36:45.782399+0000","last_fullsized":"2024-04-22T06:36:45.782399+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780266+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:36:45.780266+0000","last_peered":"2024-04-22T06:36:45.780266+0000","last_clean":"2024-04-22T06:36:45.780266+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:36:45.780266+0000","last_undegraded":"2024-04-22T06:36:45.780266+0000","last_fullsized":"2024-04-22T06:36:45.780266+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785241+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:36:45.785241+0000","last_peered":"2024-04-22T06:36:45.785241+0000","last_clean":"2024-04-22T06:36:45.785241+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:36:45.785241+0000","last_undegraded":"2024-04-22T06:36:45.785241+0000","last_fullsized":"2024-04-22T06:36:45.785241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785647+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:36:45.785647+0000","last_peered":"2024-04-22T06:36:45.785647+0000","last_clean":"2024-04-22T06:36:45.785647+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:36:45.785647+0000","last_undegraded":"2024-04-22T06:36:45.785647+0000","last_fullsized":"2024-04-22T06:36:45.785647+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785703+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:36:45.785703+0000","last_peered":"2024-04-22T06:36:45.785703+0000","last_clean":"2024-04-22T06:36:45.785703+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:36:45.785703+0000","last_undegraded":"2024-04-22T06:36:45.785703+0000","last_fullsized":"2024-04-22T06:36:45.785703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:53.092597+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:36:53.092597+0000","last_peered":"2024-04-22T06:36:53.092597+0000","last_clean":"2024-04-22T06:36:53.092597+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:36:53.092597+0000","last_undegraded":"2024-04-22T06:36:53.092597+0000","last_fullsized":"2024-04-22T06:36:53.092597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785177+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:36:45.785177+0000","last_peered":"2024-04-22T06:36:45.785177+0000","last_clean":"2024-04-22T06:36:45.785177+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:36:45.785177+0000","last_undegraded":"2024-04-22T06:36:45.785177+0000","last_fullsized":"2024-04-22T06:36:45.785177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.032174+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:36:46.032174+0000","last_peered":"2024-04-22T06:36:46.032174+0000","last_clean":"2024-04-22T06:36:46.032174+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:36:46.032174+0000","last_undegraded":"2024-04-22T06:36:46.032174+0000","last_fullsized":"2024-04-22T06:36:46.032174+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785473+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:36:45.785473+0000","last_peered":"2024-04-22T06:36:45.785473+0000","last_clean":"2024-04-22T06:36:45.785473+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:36:45.785473+0000","last_undegraded":"2024-04-22T06:36:45.785473+0000","last_fullsized":"2024-04-22T06:36:45.785473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782251+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:36:45.782251+0000","last_peered":"2024-04-22T06:36:45.782251+0000","last_clean":"2024-04-22T06:36:45.782251+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:36:45.782251+0000","last_undegraded":"2024-04-22T06:36:45.782251+0000","last_fullsized":"2024-04-22T06:36:45.782251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785814+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:36:45.785814+0000","last_peered":"2024-04-22T06:36:45.785814+0000","last_clean":"2024-04-22T06:36:45.785814+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:36:45.785814+0000","last_undegraded":"2024-04-22T06:36:45.785814+0000","last_fullsized":"2024-04-22T06:36:45.785814+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782532+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:36:45.782532+0000","last_peered":"2024-04-22T06:36:45.782532+0000","last_clean":"2024-04-22T06:36:45.782532+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:36:45.782532+0000","last_undegraded":"2024-04-22T06:36:45.782532+0000","last_fullsized":"2024-04-22T06:36:45.782532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780142+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:36:45.780142+0000","last_peered":"2024-04-22T06:36:45.780142+0000","last_clean":"2024-04-22T06:36:45.780142+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:36:45.780142+0000","last_undegraded":"2024-04-22T06:36:45.780142+0000","last_fullsized":"2024-04-22T06:36:45.780142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780125+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:36:45.780125+0000","last_peered":"2024-04-22T06:36:45.780125+0000","last_clean":"2024-04-22T06:36:45.780125+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:36:45.780125+0000","last_undegraded":"2024-04-22T06:36:45.780125+0000","last_fullsized":"2024-04-22T06:36:45.780125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031857+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:36:46.031857+0000","last_peered":"2024-04-22T06:36:46.031857+0000","last_clean":"2024-04-22T06:36:46.031857+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:36:46.031857+0000","last_undegraded":"2024-04-22T06:36:46.031857+0000","last_fullsized":"2024-04-22T06:36:46.031857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782370+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:36:45.782370+0000","last_peered":"2024-04-22T06:36:45.782370+0000","last_clean":"2024-04-22T06:36:45.782370+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:36:45.782370+0000","last_undegraded":"2024-04-22T06:36:45.782370+0000","last_fullsized":"2024-04-22T06:36:45.782370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.783162+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:36:45.783162+0000","last_peered":"2024-04-22T06:36:45.783162+0000","last_clean":"2024-04-22T06:36:45.783162+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:36:45.783162+0000","last_undegraded":"2024-04-22T06:36:45.783162+0000","last_fullsized":"2024-04-22T06:36:45.783162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780706+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:36:45.780706+0000","last_peered":"2024-04-22T06:36:45.780706+0000","last_clean":"2024-04-22T06:36:45.780706+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:36:45.780706+0000","last_undegraded":"2024-04-22T06:36:45.780706+0000","last_fullsized":"2024-04-22T06:36:45.780706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.785720+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:36:45.785720+0000","last_peered":"2024-04-22T06:36:45.785720+0000","last_clean":"2024-04-22T06:36:45.785720+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:36:45.785720+0000","last_undegraded":"2024-04-22T06:36:45.785720+0000","last_fullsized":"2024-04-22T06:36:45.785720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:46.031911+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:36:46.031911+0000","last_peered":"2024-04-22T06:36:46.031911+0000","last_clean":"2024-04-22T06:36:46.031911+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:36:46.031911+0000","last_undegraded":"2024-04-22T06:36:46.031911+0000","last_fullsized":"2024-04-22T06:36:46.031911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.784412+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:36:45.784412+0000","last_peered":"2024-04-22T06:36:45.784412+0000","last_clean":"2024-04-22T06:36:45.784412+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:36:45.784412+0000","last_undegraded":"2024-04-22T06:36:45.784412+0000","last_fullsized":"2024-04-22T06:36:45.784412+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.780917+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:36:45.780917+0000","last_peered":"2024-04-22T06:36:45.780917+0000","last_clean":"2024-04-22T06:36:45.780917+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:36:45.780917+0000","last_undegraded":"2024-04-22T06:36:45.780917+0000","last_fullsized":"2024-04-22T06:36:45.780917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782398+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:36:45.782398+0000","last_peered":"2024-04-22T06:36:45.782398+0000","last_clean":"2024-04-22T06:36:45.782398+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:36:45.782398+0000","last_undegraded":"2024-04-22T06:36:45.782398+0000","last_fullsized":"2024-04-22T06:36:45.782398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-22T06:36:45.782642+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:36:45.782642+0000","last_peered":"2024-04-22T06:36:45.782642+0000","last_clean":"2024-04-22T06:36:45.782642+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:36:45.782642+0000","last_undegraded":"2024-04-22T06:36:45.782642+0000","last_fullsized":"2024-04-22T06:36:45.782642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66278,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":709,"num_read_kb":425,"num_write":235,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":476683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514529,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1695744,"data_stored":1127500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710759,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":476545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972395,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":476923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266736,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1613824,"data_stored":1069441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430262,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":476262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":536640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1024000,"data_stored":477601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":536792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":476087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888127,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":476514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691897,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32772,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712380,"statfs":{"total":95995035648,"available":95961477120,"internally_reserved":0,"allocated":1626112,"data_stored":1068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":61369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:36:57.746 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T06:36:57.747 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:36:57.747 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:36:57.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T06:36:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:57 smithi112 ceph-mon[31236]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:36:57.984 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:36:57.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:57 smithi177 ceph-mon[31346]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:36:58.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:57 smithi105 ceph-mon[27916]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 6 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:36:58.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:58 smithi112 ceph-mon[31236]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:58.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:58 smithi177 ceph-mon[31346]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:59.048 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:58 smithi105 ceph-mon[27916]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 -- 172.21.15.105:0/3658654633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 msgr2=0x7fe90410f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 --2- 172.21.15.105:0/3658654633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe90410f920 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00b0a0 tx=0x7fe8fc00b3b0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 -- 172.21.15.105:0/3658654633 shutdown_connections 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 --2- 172.21.15.105:0/3658654633 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe90410f920 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 --2- 172.21.15.105:0/3658654633 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe904108c20 0x7fe904109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 --2- 172.21.15.105:0/3658654633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe90406b110 0x7fe90406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 -- 172.21.15.105:0/3658654633 >> 172.21.15.105:0/3658654633 conn(0x7fe9040788f0 msgr2=0x7fe904078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 -- 172.21.15.105:0/3658654633 shutdown_connections 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 -- 172.21.15.105:0/3658654633 wait complete. 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.533+0000 7fe909802700 1 Processor -- start 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 -- start start 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90406b110 0x7fe904199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:59.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe904108c20 0x7fe90419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112c60 con 0x7fe90406bad0 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112ae0 con 0x7fe904108c20 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe909802700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe904112de0 con 0x7fe90406b110 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52834/0 (socket says 172.21.15.105:52834) 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 -- 172.21.15.105:0/1076963230 learned_addr learned my addr 172.21.15.105:0/1076963230 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe9037fe700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe904108c20 0x7fe90419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe902ffd700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90406b110 0x7fe904199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 -- 172.21.15.105:0/1076963230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90406b110 msgr2=0x7fe904199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90406b110 0x7fe904199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 -- 172.21.15.105:0/1076963230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe904108c20 msgr2=0x7fe90419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe904108c20 0x7fe90419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.534+0000 7fe8fa7fc700 1 -- 172.21.15.105:0/1076963230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90419e020 con 0x7fe90406bad0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe8fa7fc700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d950 tx=0x7fe8ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec016070 con 0x7fe90406bad0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90419e310 con 0x7fe90406bad0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8ec00fd40 con 0x7fe90406bad0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00e3b0 con 0x7fe90406bad0 2024-04-22T06:36:59.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.535+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9041a5b90 con 0x7fe90406bad0 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.536+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe904104190 con 0x7fe90406bad0 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.537+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe8ec019070 con 0x7fe90406bad0 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.538+0000 7fe908800700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 0x7fe8f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.538+0000 7fe902ffd700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 0x7fe8f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.538+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fe8ec0cff40 con 0x7fe90406bad0 2024-04-22T06:36:59.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.538+0000 7fe902ffd700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 0x7fe8f00a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4004630 tx=0x7fe8f400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:36:59.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.542+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8ec09c8c0 con 0x7fe90406bad0 2024-04-22T06:36:59.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.786+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe90402df80 con 0x7fe90406bad0 2024-04-22T06:36:59.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.786+0000 7fe908800700 1 -- 172.21.15.105:0/1076963230 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe8ec09c1c0 con 0x7fe90406bad0 2024-04-22T06:36:59.786 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:36:59.786 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.788+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 msgr2=0x7fe8f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.788+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 0x7fe8f00a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4004630 tx=0x7fe8f400b3c0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.788+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 msgr2=0x7fe904199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.788+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00d950 tx=0x7fe8ec00dc60 comp rx=0 tx=0).stop 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 shutdown_connections 2024-04-22T06:36:59.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fe8f00a3cd0 0x7fe8f00a6190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe904108c20 0x7fe90419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe90406b110 0x7fe904199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 --2- 172.21.15.105:0/1076963230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe90406bad0 0x7fe904199550 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 >> 172.21.15.105:0/1076963230 conn(0x7fe9040788f0 msgr2=0x7fe904100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 shutdown_connections 2024-04-22T06:36:59.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:36:59.789+0000 7fe909802700 1 -- 172.21.15.105:0/1076963230 wait complete. 2024-04-22T06:36:59.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:36:59 smithi112 ceph-mon[31236]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:36:59.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:36:59 smithi177 ceph-mon[31346]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:37:00.163 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:36:59 smithi105 ceph-mon[27916]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-04-22T06:37:00.491 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:37:00.491 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:00.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/2985754735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac100e70 msgr2=0x7fa4ac101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/2985754735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac100e70 0x7fa4ac101290 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009900 tx=0x7fa4a002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:00.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/2985754735 shutdown_connections 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/2985754735 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac102a10 0x7fa4ac109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/2985754735 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4ac102050 0x7fa4ac1024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/2985754735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac100e70 0x7fa4ac101290 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/2985754735 >> 172.21.15.105:0/2985754735 conn(0x7fa4ac0fc470 msgr2=0x7fa4ac0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/2985754735 shutdown_connections 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/2985754735 wait complete. 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 Processor -- start 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 -- start start 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.609+0000 7fa4b2ef4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 0x7fa4ac06b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4ac102a10 0x7fa4ac19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac10ca00 con 0x7fa4ac102050 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac10c880 con 0x7fa4ac100e70 2024-04-22T06:37:00.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac10cb80 con 0x7fa4ac102a10 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52846/0 (socket says 172.21.15.105:52846) 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 -- 172.21.15.105:0/527816684 learned_addr learned my addr 172.21.15.105:0/527816684 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b0c90640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 0x7fa4ac06b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 -- 172.21.15.105:0/527816684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4ac102a10 msgr2=0x7fa4ac19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4ac102a10 0x7fa4ac19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 -- 172.21.15.105:0/527816684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 msgr2=0x7fa4ac06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 0x7fa4ac06b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 -- 172.21.15.105:0/527816684 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac19fe50 con 0x7fa4ac102050 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b0c90640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 0x7fa4ac06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4abfff640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa49800d770 tx=0x7fa49800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498017070 con 0x7fa4ac102050 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4980054d0 con 0x7fa4ac102050 2024-04-22T06:37:00.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ac1a00e0 con 0x7fa4ac102050 2024-04-22T06:37:00.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498004c30 con 0x7fa4ac102050 2024-04-22T06:37:00.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.610+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ac1a05e0 con 0x7fa4ac102050 2024-04-22T06:37:00.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.611+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ac108260 con 0x7fa4ac102050 2024-04-22T06:37:00.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.612+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa498004e10 con 0x7fa4ac102050 2024-04-22T06:37:00.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.613+0000 7fa4a9ffb640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 0x7fa49c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.613+0000 7fa4b0c90640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 0x7fa49c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.614+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fa4980d0000 con 0x7fa4ac102050 2024-04-22T06:37:00.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.614+0000 7fa4b0c90640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 0x7fa49c0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00096c0 tx=0x7fa4a0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:00.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.618+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa49809c980 con 0x7fa4ac102050 2024-04-22T06:37:00.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.803+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4ac02d7c0 con 0x7fa4ac102050 2024-04-22T06:37:00.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.804+0000 7fa4a9ffb640 1 -- 172.21.15.105:0/527816684 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7fa49809c280 con 0x7fa4ac102050 2024-04-22T06:37:00.804 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:00.804 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:00.806 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.806+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 msgr2=0x7fa49c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.806 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.806+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 0x7fa49c0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00096c0 tx=0x7fa4a0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.806 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.806+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 msgr2=0x7fa4ac19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.806+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa49800d770 tx=0x7fa49800dc30 comp rx=0 tx=0).stop 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 shutdown_connections 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa49c0a3c40 0x7fa49c0a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa4ac102a10 0x7fa4ac19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4ac102050 0x7fa4ac19cf20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 --2- 172.21.15.105:0/527816684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4ac100e70 0x7fa4ac06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 >> 172.21.15.105:0/527816684 conn(0x7fa4ac0fc470 msgr2=0x7fa4ac105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 shutdown_connections 2024-04-22T06:37:00.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.807+0000 7fa4b2ef4640 1 -- 172.21.15.105:0/527816684 wait complete. 2024-04-22T06:37:00.818 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:00 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1076963230' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:37:00.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:00.820 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:00.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 -- 172.21.15.105:0/945053952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 msgr2=0x7f5b0c101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 --2- 172.21.15.105:0/945053952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c101560 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009900 tx=0x7f5afc02bc80 comp rx=0 tx=0).stop 2024-04-22T06:37:00.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 -- 172.21.15.105:0/945053952 shutdown_connections 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 --2- 172.21.15.105:0/945053952 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 0x7f5b0c109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 --2- 172.21.15.105:0/945053952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c101560 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 --2- 172.21.15.105:0/945053952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 -- 172.21.15.105:0/945053952 >> 172.21.15.105:0/945053952 conn(0x7f5b0c0fbb90 msgr2=0x7f5b0c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 -- 172.21.15.105:0/945053952 shutdown_connections 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.922+0000 7f5b133d5640 1 -- 172.21.15.105:0/945053952 wait complete. 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 Processor -- start 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 -- start start 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 0x7f5b0c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10c9f0 con 0x7f5b0c1010e0 2024-04-22T06:37:00.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10c870 con 0x7f5b0c102520 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b133d5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0c10cb70 con 0x7f5b0c101aa0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52878/0 (socket says 172.21.15.105:52878) 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 -- 172.21.15.105:0/1247668513 learned_addr learned my addr 172.21.15.105:0/1247668513 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11972640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b10970640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 0x7f5b0c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 -- 172.21.15.105:0/1247668513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 msgr2=0x7f5b0c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 0x7f5b0c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 -- 172.21.15.105:0/1247668513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 msgr2=0x7f5b0c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11171640 1 -- 172.21.15.105:0/1247668513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0c19fe60 con 0x7f5b0c1010e0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.923+0000 7f5b11972640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b11171640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c890 tx=0x7f5b0800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:00.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08004220 con 0x7f5b0c1010e0 2024-04-22T06:37:00.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b080043c0 con 0x7f5b0c1010e0 2024-04-22T06:37:00.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0c1a0150 con 0x7f5b0c1010e0 2024-04-22T06:37:00.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08004a00 con 0x7f5b0c1010e0 2024-04-22T06:37:00.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.924+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0c1a0670 con 0x7f5b0c1010e0 2024-04-22T06:37:00.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.925+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b0c1a0930 con 0x7f5b0c1010e0 2024-04-22T06:37:00.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.926+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5b080079b0 con 0x7f5b0c1010e0 2024-04-22T06:37:00.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.926+0000 7f5b027fc640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 0x7f5af80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:00.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.926+0000 7f5b10970640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 0x7f5af80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:00.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.927+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5b080d0690 con 0x7f5b0c1010e0 2024-04-22T06:37:00.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.927+0000 7f5b10970640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 0x7f5af80a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5afc006010 tx=0x7f5afc005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:00.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:00.930+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b08014030 con 0x7f5b0c1010e0 2024-04-22T06:37:00.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:00 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1076963230' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:37:00.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:00 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1076963230' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T06:37:01.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.108+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b0c1a0b00 con 0x7f5b0c1010e0 2024-04-22T06:37:01.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.109+0000 7f5b027fc640 1 -- 172.21.15.105:0/1247668513 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f5b0809d010 con 0x7f5b0c1010e0 2024-04-22T06:37:01.110 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:01.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 msgr2=0x7f5af80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 0x7f5af80a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5afc006010 tx=0x7f5afc005be0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 msgr2=0x7f5b0c198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c890 tx=0x7f5b0800cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:01.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 shutdown_connections 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5af80a3c40 0x7f5af80a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b0c102520 0x7f5b0c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5b0c101aa0 0x7f5b0c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 --2- 172.21.15.105:0/1247668513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5b0c1010e0 0x7f5b0c198240 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 >> 172.21.15.105:0/1247668513 conn(0x7f5b0c0fbb90 msgr2=0x7f5b0c0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 shutdown_connections 2024-04-22T06:37:01.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.111+0000 7f5b133d5640 1 -- 172.21.15.105:0/1247668513 wait complete. 2024-04-22T06:37:01.121 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 -- 172.21.15.105:0/2043477803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103b40 msgr2=0x7f78a810a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 --2- 172.21.15.105:0/2043477803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103b40 0x7f78a810a3d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f78a0009ca0 tx=0x7f78a002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 -- 172.21.15.105:0/2043477803 shutdown_connections 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 --2- 172.21.15.105:0/2043477803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103b40 0x7f78a810a3d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 --2- 172.21.15.105:0/2043477803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103180 0x7f78a8103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 --2- 172.21.15.105:0/2043477803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 0x7f78a81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.228 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 -- 172.21.15.105:0/2043477803 >> 172.21.15.105:0/2043477803 conn(0x7f78a80fd500 msgr2=0x7f78a80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 -- 172.21.15.105:0/2043477803 shutdown_connections 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.226+0000 7f78ae796640 1 -- 172.21.15.105:0/2043477803 wait complete. 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 Processor -- start 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 -- start start 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 0x7f78a8198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 0x7f78a819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810db30 con 0x7f78a8103180 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810d9b0 con 0x7f78a8103b40 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78ae796640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78a810dcb0 con 0x7f78a8101f90 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78a77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78a77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52890/0 (socket says 172.21.15.105:52890) 2024-04-22T06:37:01.229 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78a77fe640 1 -- 172.21.15.105:0/955579516 learned_addr learned my addr 172.21.15.105:0/955579516 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78acd33640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 0x7f78a819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78a7fff640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 0x7f78a8198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.227+0000 7f78a77fe640 1 -- 172.21.15.105:0/955579516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 msgr2=0x7f78a8198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a77fe640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 0x7f78a8198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a77fe640 1 -- 172.21.15.105:0/955579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 msgr2=0x7f78a819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a77fe640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 0x7f78a819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a77fe640 1 -- 172.21.15.105:0/955579516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a819fe50 con 0x7f78a8103180 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78acd33640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 0x7f78a819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a77fe640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f789c00c890 tx=0x7f789c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c004220 con 0x7f78a8103180 2024-04-22T06:37:01.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c0043c0 con 0x7f78a8103180 2024-04-22T06:37:01.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a81a0140 con 0x7f78a8103180 2024-04-22T06:37:01.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c004a00 con 0x7f78a8103180 2024-04-22T06:37:01.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.228+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a81a0690 con 0x7f78a8103180 2024-04-22T06:37:01.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.229+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a8109320 con 0x7f78a8103180 2024-04-22T06:37:01.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.230+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f789c0079b0 con 0x7f78a8103180 2024-04-22T06:37:01.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.230+0000 7f78a57fa640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 0x7f78900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.230+0000 7f78a7fff640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 0x7f78900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.231+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f789c0d0280 con 0x7f78a8103180 2024-04-22T06:37:01.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.231+0000 7f78a7fff640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 0x7f78900a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f78a8102fe0 tx=0x7f7898008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.235+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f789c014030 con 0x7f78a8103180 2024-04-22T06:37:01.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.414+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78a802d810 con 0x7f78a8103180 2024-04-22T06:37:01.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.414+0000 7f78a57fa640 1 -- 172.21.15.105:0/955579516 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f789c09cc00 con 0x7f78a8103180 2024-04-22T06:37:01.414 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:01.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 msgr2=0x7f78900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 0x7f78900a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f78a8102fe0 tx=0x7f7898008040 comp rx=0 tx=0).stop 2024-04-22T06:37:01.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 msgr2=0x7f78a8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f789c00c890 tx=0x7f789c00cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:01.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 shutdown_connections 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f78900a3c40 0x7f78900a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a8103b40 0x7f78a819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f78a8103180 0x7f78a8198770 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 --2- 172.21.15.105:0/955579516 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a8101f90 0x7f78a8198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 >> 172.21.15.105:0/955579516 conn(0x7f78a80fd500 msgr2=0x7f78a81064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 shutdown_connections 2024-04-22T06:37:01.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.416+0000 7f78ae796640 1 -- 172.21.15.105:0/955579516 wait complete. 2024-04-22T06:37:01.427 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:01.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 -- 172.21.15.105:0/3222155449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0881029a0 msgr2=0x7fc0881092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 --2- 172.21.15.105:0/3222155449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0881029a0 0x7fc0881092c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc080009ca0 tx=0x7fc08002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 -- 172.21.15.105:0/3222155449 shutdown_connections 2024-04-22T06:37:01.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 --2- 172.21.15.105:0/3222155449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc0881029a0 0x7fc0881092c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 --2- 172.21.15.105:0/3222155449 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc088101fe0 0x7fc088102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 --2- 172.21.15.105:0/3222155449 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088100e90 0x7fc0881012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.528+0000 7fc08f946640 1 -- 172.21.15.105:0/3222155449 >> 172.21.15.105:0/3222155449 conn(0x7fc0880fc400 msgr2=0x7fc0880fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- 172.21.15.105:0/3222155449 shutdown_connections 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- 172.21.15.105:0/3222155449 wait complete. 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 Processor -- start 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- start start 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088101fe0 0x7fc08819d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0881029a0 0x7fc08819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810cae0 con 0x7fc088100e90 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810c960 con 0x7fc0881029a0 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08f946640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc08810cc60 con 0x7fc088101fe0 2024-04-22T06:37:01.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08d6e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08d6e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52918/0 (socket says 172.21.15.105:52918) 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08d6e2640 1 -- 172.21.15.105:0/2489544254 learned_addr learned my addr 172.21.15.105:0/2489544254 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.529+0000 7fc08dee3640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0881029a0 0x7fc08819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08cee1640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088101fe0 0x7fc08819d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 -- 172.21.15.105:0/2489544254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088101fe0 msgr2=0x7fc08819d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088101fe0 0x7fc08819d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 -- 172.21.15.105:0/2489544254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0881029a0 msgr2=0x7fc08819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0881029a0 0x7fc08819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 -- 172.21.15.105:0/2489544254 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08819ff60 con 0x7fc088100e90 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08d6e2640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc07800b4a0 tx=0x7fc07800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078014070 con 0x7fc088100e90 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc078010ac0 con 0x7fc088100e90 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0881a01f0 con 0x7fc088100e90 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078017c30 con 0x7fc088100e90 2024-04-22T06:37:01.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.530+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0881a0770 con 0x7fc088100e90 2024-04-22T06:37:01.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.531+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc088108210 con 0x7fc088100e90 2024-04-22T06:37:01.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.532+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc078017070 con 0x7fc088100e90 2024-04-22T06:37:01.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.532+0000 7fc07e7fc640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 0x7fc0740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.533+0000 7fc08cee1640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 0x7fc0740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.533+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc0780cf720 con 0x7fc088100e90 2024-04-22T06:37:01.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.533+0000 7fc08cee1640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 0x7fc0740a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc0880699c0 tx=0x7fc084008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.537+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc07809c1b0 con 0x7fc088100e90 2024-04-22T06:37:01.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.717+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc08802d810 con 0x7fc088100e90 2024-04-22T06:37:01.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.717+0000 7fc07e7fc640 1 -- 172.21.15.105:0/2489544254 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fc07809bab0 con 0x7fc088100e90 2024-04-22T06:37:01.717 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 msgr2=0x7fc0740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 0x7fc0740a6100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc0880699c0 tx=0x7fc084008040 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 msgr2=0x7fc088069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc07800b4a0 tx=0x7fc07800b960 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 shutdown_connections 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0740a3c40 0x7fc0740a6100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc0881029a0 0x7fc08819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc088101fe0 0x7fc08819d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 --2- 172.21.15.105:0/2489544254 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc088100e90 0x7fc088069c50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 >> 172.21.15.105:0/2489544254 conn(0x7fc0880fc400 msgr2=0x7fc0881053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 shutdown_connections 2024-04-22T06:37:01.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.719+0000 7fc08f946640 1 -- 172.21.15.105:0/2489544254 wait complete. 2024-04-22T06:37:01.730 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 -- 172.21.15.105:0/2060285923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 msgr2=0x7f8ff8103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 --2- 172.21.15.105:0/2060285923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff8103600 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009900 tx=0x7f8fe802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 -- 172.21.15.105:0/2060285923 shutdown_connections 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 --2- 172.21.15.105:0/2060285923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ff8103b40 0x7f8ff810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 --2- 172.21.15.105:0/2060285923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff8103600 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 --2- 172.21.15.105:0/2060285923 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8101f90 0x7f8ff81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 -- 172.21.15.105:0/2060285923 >> 172.21.15.105:0/2060285923 conn(0x7f8ff80fd500 msgr2=0x7f8ff80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 -- 172.21.15.105:0/2060285923 shutdown_connections 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.829+0000 7f900005c640 1 -- 172.21.15.105:0/2060285923 wait complete. 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 Processor -- start 2024-04-22T06:37:01.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 -- start start 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ff8101f90 0x7f8ff8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 0x7f8ff819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810db60 con 0x7f8ff8103180 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810d9e0 con 0x7f8ff8101f90 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f900005c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810dce0 con 0x7f8ff8103b40 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52938/0 (socket says 172.21.15.105:52938) 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 -- 172.21.15.105:0/2852344875 learned_addr learned my addr 172.21.15.105:0/2852344875 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffe5f9640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 0x7f8ff819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 -- 172.21.15.105:0/2852344875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 msgr2=0x7f8ff819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 0x7f8ff819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 -- 172.21.15.105:0/2852344875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ff8101f90 msgr2=0x7f8ff8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:01.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ff8101f90 0x7f8ff8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 -- 172.21.15.105:0/2852344875 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff819fdf0 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffe5f9640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 0x7f8ff819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8ffd5f7640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.830+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8043020 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.831+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe80054c0 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.831+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a0080 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.831+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003db0 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.831+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a06a0 con 0x7f8ff8103180 2024-04-22T06:37:01.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.832+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8109250 con 0x7f8ff8103180 2024-04-22T06:37:01.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.833+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8fe8046020 con 0x7f8ff8103180 2024-04-22T06:37:01.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.833+0000 7f8feeffd640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 0x7f8fe40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:01.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.833+0000 7f8ffddf8640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 0x7f8fe40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:01.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.833+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f8fe80f2b50 con 0x7f8ff8103180 2024-04-22T06:37:01.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.834+0000 7f8ffddf8640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 0x7f8fe40a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8102fe0 tx=0x7f8ff4005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:01.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:01.838+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fe804d020 con 0x7f8ff8103180 2024-04-22T06:37:01.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:01 smithi112 ceph-mon[31236]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:01.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:01 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/527816684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:01.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:01 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1247668513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:01.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:01 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/955579516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:01 smithi177 ceph-mon[31346]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:01 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/527816684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:01 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1247668513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:01.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:01 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/955579516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.052+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ff802d850 con 0x7f8ff8103180 2024-04-22T06:37:02.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.053+0000 7f8feeffd640 1 -- 172.21.15.105:0/2852344875 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7f8fe8004c20 con 0x7f8ff8103180 2024-04-22T06:37:02.052 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:02.052 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:37:02.052 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:02.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.054+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 msgr2=0x7f8fe40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.054+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 0x7f8fe40a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8102fe0 tx=0x7f8ff4005c90 comp rx=0 tx=0).stop 2024-04-22T06:37:02.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 msgr2=0x7f8ff81987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80042f0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 shutdown_connections 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8fe40a3cc0 0x7f8fe40a6180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ff8103b40 0x7f8ff819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ff8103180 0x7f8ff81987b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 --2- 172.21.15.105:0/2852344875 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ff8101f90 0x7f8ff8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 >> 172.21.15.105:0/2852344875 conn(0x7f8ff80fd500 msgr2=0x7f8ff8106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 shutdown_connections 2024-04-22T06:37:02.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.055+0000 7f900005c640 1 -- 172.21.15.105:0/2852344875 wait complete. 2024-04-22T06:37:02.064 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:01 smithi105 ceph-mon[27916]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:02.064 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/527816684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:02.064 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1247668513' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.065 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:01 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/955579516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.066 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 -- 172.21.15.105:0/1737830634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4101fd0 msgr2=0x7fc6d4102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/1737830634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4101fd0 0x7fc6d4102450 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009900 tx=0x7fc6d002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 -- 172.21.15.105:0/1737830634 shutdown_connections 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/1737830634 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d4102990 0x7fc6d41092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/1737830634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4101fd0 0x7fc6d4102450 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/1737830634 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6d4100e80 0x7fc6d41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 -- 172.21.15.105:0/1737830634 >> 172.21.15.105:0/1737830634 conn(0x7fc6d40fc410 msgr2=0x7fc6d40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 -- 172.21.15.105:0/1737830634 shutdown_connections 2024-04-22T06:37:02.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.165+0000 7fc6dbb50640 1 -- 172.21.15.105:0/1737830634 wait complete. 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 Processor -- start 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 -- start start 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d4101fd0 0x7fc6d41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6d4102990 0x7fc6d419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410db60 con 0x7fc6d4100e80 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d9e0 con 0x7fc6d4102990 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6dbb50640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410dce0 con 0x7fc6d4101fd0 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52966/0 (socket says 172.21.15.105:52966) 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 -- 172.21.15.105:0/3800649057 learned_addr learned my addr 172.21.15.105:0/3800649057 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 -- 172.21.15.105:0/3800649057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d4101fd0 msgr2=0x7fc6d41987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d4101fd0 0x7fc6d41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 -- 172.21.15.105:0/3800649057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6d4102990 msgr2=0x7fc6d419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6d4102990 0x7fc6d419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.166+0000 7fc6d98ec640 1 -- 172.21.15.105:0/3800649057 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d419fe80 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6d98ec640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400b4a0 tx=0x7fc6c400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4014070 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c4010ac0 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d41a0170 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4017c30 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.167+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d41a0740 con 0x7fc6d4100e80 2024-04-22T06:37:02.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.168+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d4108200 con 0x7fc6d4100e80 2024-04-22T06:37:02.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.169+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fc6c4017070 con 0x7fc6d4100e80 2024-04-22T06:37:02.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.169+0000 7fc6caffd640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 0x7fc6c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.169+0000 7fc6d90eb640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 0x7fc6c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.169+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc6c40cfb10 con 0x7fc6d4100e80 2024-04-22T06:37:02.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.170+0000 7fc6d90eb640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 0x7fc6c00a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0036040 tx=0x7fc6d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.174+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6c409c5a0 con 0x7fc6d4100e80 2024-04-22T06:37:02.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.354+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6d402d800 con 0x7fc6d4100e80 2024-04-22T06:37:02.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.356+0000 7fc6caffd640 1 -- 172.21.15.105:0/3800649057 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7fc6c409bea0 con 0x7fc6d4100e80 2024-04-22T06:37:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:02.354 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:02.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 msgr2=0x7fc6c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 0x7fc6c00a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0036040 tx=0x7fc6d0000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 msgr2=0x7fc6d4198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400b4a0 tx=0x7fc6c400b960 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 shutdown_connections 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc6c00a3c40 0x7fc6c00a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6d4102990 0x7fc6d419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc6d4101fd0 0x7fc6d41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 --2- 172.21.15.105:0/3800649057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6d4100e80 0x7fc6d4198260 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 >> 172.21.15.105:0/3800649057 conn(0x7fc6d40fc410 msgr2=0x7fc6d41053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 shutdown_connections 2024-04-22T06:37:02.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.358+0000 7fc6dbb50640 1 -- 172.21.15.105:0/3800649057 wait complete. 2024-04-22T06:37:02.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:02.368 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:02.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/1330565711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 msgr2=0x7f5fc410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/1330565711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc410a0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009ca0 tx=0x7f5fbc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/1330565711 shutdown_connections 2024-04-22T06:37:02.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/1330565711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc410a0c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/1330565711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc41053a0 0x7f5fc4107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/1330565711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc4102a70 0x7f5fc4104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/1330565711 >> 172.21.15.105:0/1330565711 conn(0x7f5fc40fc400 msgr2=0x7f5fc40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/1330565711 shutdown_connections 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.471+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/1330565711 wait complete. 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 Processor -- start 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 -- start start 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 0x7f5fc4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 0x7f5fc4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410dac0 con 0x7f5fc4107cd0 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410d940 con 0x7f5fc41053a0 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410dc40 con 0x7f5fc4102a70 2024-04-22T06:37:02.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52990/0 (socket says 172.21.15.105:52990) 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 -- 172.21.15.105:0/966711086 learned_addr learned my addr 172.21.15.105:0/966711086 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc8f52640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 0x7f5fc4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 -- 172.21.15.105:0/966711086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 msgr2=0x7f5fc4198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc3fff640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 0x7f5fc4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 0x7f5fc4198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 -- 172.21.15.105:0/966711086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 msgr2=0x7f5fc4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 0x7f5fc4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 -- 172.21.15.105:0/966711086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc419fe00 con 0x7f5fc4107cd0 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc8f52640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 0x7f5fc4198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc3fff640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 0x7f5fc4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:02.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc9753640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc02c700 tx=0x7f5fbc0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc041020 con 0x7f5fc4107cd0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.472+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc41a0090 con 0x7f5fc4107cd0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.473+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc41a05d0 con 0x7f5fc4107cd0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.473+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fbc034870 con 0x7f5fc4107cd0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.473+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc039040 con 0x7f5fc4107cd0 2024-04-22T06:37:02.475 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.474+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa80052c0 con 0x7f5fc4107cd0 2024-04-22T06:37:02.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.475+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5fbc044020 con 0x7f5fc4107cd0 2024-04-22T06:37:02.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.475+0000 7f5fc1ffb640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.475+0000 7f5fc8f52640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.476+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5fbc0f1c80 con 0x7f5fc4107cd0 2024-04-22T06:37:02.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.476+0000 7f5fc8f52640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 0x7f5fb40a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009c20 tx=0x7f5fb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.480+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fbc04b030 con 0x7f5fc4107cd0 2024-04-22T06:37:02.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.659+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fa8005880 con 0x7f5fc4107cd0 2024-04-22T06:37:02.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.659+0000 7f5fc1ffb640 1 -- 172.21.15.105:0/966711086 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f5fbc0be710 con 0x7f5fc4107cd0 2024-04-22T06:37:02.658 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 msgr2=0x7f5fb40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 0x7f5fb40a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009c20 tx=0x7f5fb8005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 msgr2=0x7f5fc419f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc02c700 tx=0x7f5fbc0093f0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 shutdown_connections 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fc4107cd0 0x7f5fc419f790 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fc41053a0 0x7f5fc4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 --2- 172.21.15.105:0/966711086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5fc4102a70 0x7f5fc4198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 >> 172.21.15.105:0/966711086 conn(0x7f5fc40fc400 msgr2=0x7f5fc41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 shutdown_connections 2024-04-22T06:37:02.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.661+0000 7f5fcb1b6640 1 -- 172.21.15.105:0/966711086 wait complete. 2024-04-22T06:37:02.671 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 -- 172.21.15.105:0/851174477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 msgr2=0x7fd964101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 --2- 172.21.15.105:0/851174477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964101280 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd960009900 tx=0x7fd96002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 -- 172.21.15.105:0/851174477 shutdown_connections 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 --2- 172.21.15.105:0/851174477 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102a00 0x7fd964109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 --2- 172.21.15.105:0/851174477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102040 0x7fd9641024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 --2- 172.21.15.105:0/851174477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964101280 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 -- 172.21.15.105:0/851174477 >> 172.21.15.105:0/851174477 conn(0x7fd9640fc440 msgr2=0x7fd9640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 -- 172.21.15.105:0/851174477 shutdown_connections 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.775+0000 7fd96c03f640 1 -- 172.21.15.105:0/851174477 wait complete. 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 Processor -- start 2024-04-22T06:37:02.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 -- start start 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102040 0x7fd9641034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102a00 0x7fd9641039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410cad0 con 0x7fd964100e60 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410c950 con 0x7fd964102a00 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96c03f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd96410cc50 con 0x7fd964102040 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53010/0 (socket says 172.21.15.105:53010) 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 -- 172.21.15.105:0/2919073211 learned_addr learned my addr 172.21.15.105:0/2919073211 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd9695da640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102040 0x7fd9641034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 -- 172.21.15.105:0/2919073211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102040 msgr2=0x7fd9641034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102040 0x7fd9641034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 -- 172.21.15.105:0/2919073211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102a00 msgr2=0x7fd9641039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd96a5dc640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102a00 0x7fd9641039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102a00 0x7fd9641039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 -- 172.21.15.105:0/2919073211 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd964104280 con 0x7fd964100e60 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.776+0000 7fd969ddb640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd960036040 tx=0x7fd9600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.777+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960043020 con 0x7fd964100e60 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.777+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd960004aa0 con 0x7fd964100e60 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.777+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960003de0 con 0x7fd964100e60 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.777+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9641a24e0 con 0x7fd964100e60 2024-04-22T06:37:02.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.777+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641a2a60 con 0x7fd964100e60 2024-04-22T06:37:02.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.778+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9641a2c60 con 0x7fd964100e60 2024-04-22T06:37:02.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.779+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd960046020 con 0x7fd964100e60 2024-04-22T06:37:02.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.779+0000 7fd95affd640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 0x7fd9500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:02.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.779+0000 7fd9695da640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 0x7fd9500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:02.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.779+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fd9600f2bb0 con 0x7fd964100e60 2024-04-22T06:37:02.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.780+0000 7fd9695da640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 0x7fd9500a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd964104be0 tx=0x7fd954008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:02.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.783+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd96004d020 con 0x7fd964100e60 2024-04-22T06:37:02.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.960+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd9641a2e30 con 0x7fd964100e60 2024-04-22T06:37:02.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.961+0000 7fd95affd640 1 -- 172.21.15.105:0/2919073211 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fd9600bf640 con 0x7fd964100e60 2024-04-22T06:37:02.960 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:02.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 msgr2=0x7fd9500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 0x7fd9500a6180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd964104be0 tx=0x7fd954008040 comp rx=0 tx=0).stop 2024-04-22T06:37:02.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 msgr2=0x7fd964104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:02.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd960036040 tx=0x7fd9600042f0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 shutdown_connections 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fd9500a3cc0 0x7fd9500a6180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd964102a00 0x7fd9641039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd964102040 0x7fd9641034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 --2- 172.21.15.105:0/2919073211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd964100e60 0x7fd964104e70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 >> 172.21.15.105:0/2919073211 conn(0x7fd9640fc440 msgr2=0x7fd96406a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 shutdown_connections 2024-04-22T06:37:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:02.963+0000 7fd96c03f640 1 -- 172.21.15.105:0/2919073211 wait complete. 2024-04-22T06:37:02.971 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:02 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2489544254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.971 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:02 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2852344875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:02.971 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:02 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3800649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:02.971 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:02 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/966711086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:02.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:02 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2489544254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:02.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:02 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2852344875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:02.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:02 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3800649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:02.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:02 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/966711086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 -- 172.21.15.105:0/80835285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102a00 msgr2=0x7fa7cc109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 --2- 172.21.15.105:0/80835285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102a00 0x7fa7cc109320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009ca0 tx=0x7fa7c402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 -- 172.21.15.105:0/80835285 shutdown_connections 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 --2- 172.21.15.105:0/80835285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102a00 0x7fa7cc109320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 --2- 172.21.15.105:0/80835285 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc102040 0x7fa7cc1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 --2- 172.21.15.105:0/80835285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc100e60 0x7fa7cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 -- 172.21.15.105:0/80835285 >> 172.21.15.105:0/80835285 conn(0x7fa7cc0fc440 msgr2=0x7fa7cc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 -- 172.21.15.105:0/80835285 shutdown_connections 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.075+0000 7fa7d2315640 1 -- 172.21.15.105:0/80835285 wait complete. 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 Processor -- start 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 -- start start 2024-04-22T06:37:03.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc100e60 0x7fa7cc198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc102a00 0x7fa7cc19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10db30 con 0x7fa7cc102040 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10d9b0 con 0x7fa7cc102a00 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d2315640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc10dcb0 con 0x7fa7cc100e60 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53028/0 (socket says 172.21.15.105:53028) 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 -- 172.21.15.105:0/2123169465 learned_addr learned my addr 172.21.15.105:0/2123169465 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7d08b2640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc102a00 0x7fa7cc19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cbfff640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc100e60 0x7fa7cc198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 -- 172.21.15.105:0/2123169465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc100e60 msgr2=0x7fa7cc198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc100e60 0x7fa7cc198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 -- 172.21.15.105:0/2123169465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc102a00 msgr2=0x7fa7cc19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc102a00 0x7fa7cc19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.076+0000 7fa7cb7fe640 1 -- 172.21.15.105:0/2123169465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc19fd70 con 0x7fa7cc102040 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7cb7fe640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000c920 tx=0x7fa7c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0007c70 con 0x7fa7cc102040 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7c0007e10 con 0x7fa7cc102040 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7cc1a0060 con 0x7fa7cc102040 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0018820 con 0x7fa7cc102040 2024-04-22T06:37:03.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.077+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7cc1a05b0 con 0x7fa7cc102040 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.078+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7cc1081a0 con 0x7fa7cc102040 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.079+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa7c000f450 con 0x7fa7cc102040 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.079+0000 7fa7c97fa640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 0x7fa7b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.079+0000 7fa7cbfff640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 0x7fa7b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.079+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fa7c00d05a0 con 0x7fa7cc102040 2024-04-22T06:37:03.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.080+0000 7fa7cbfff640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 0x7fa7b40a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc005fd0 tx=0x7fa7bc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.084+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7c0014030 con 0x7fa7cc102040 2024-04-22T06:37:03.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:02 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2489544254' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:02 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2852344875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:03.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:02 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3800649057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:03.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:02 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/966711086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.261+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa7cc02d5c0 con 0x7fa7cc102040 2024-04-22T06:37:03.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.261+0000 7fa7c97fa640 1 -- 172.21.15.105:0/2123169465 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fa7c009cf20 con 0x7fa7cc102040 2024-04-22T06:37:03.260 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 msgr2=0x7fa7b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 0x7fa7b40a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc005fd0 tx=0x7fa7bc005850 comp rx=0 tx=0).stop 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 msgr2=0x7fa7cc198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000c920 tx=0x7fa7c000cde0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 shutdown_connections 2024-04-22T06:37:03.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa7b40a3c40 0x7fa7b40a6100 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7cc102a00 0x7fa7cc19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7cc102040 0x7fa7cc198690 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 --2- 172.21.15.105:0/2123169465 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7cc100e60 0x7fa7cc198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 >> 172.21.15.105:0/2123169465 conn(0x7fa7cc0fc440 msgr2=0x7fa7cc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 shutdown_connections 2024-04-22T06:37:03.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.263+0000 7fa7d2315640 1 -- 172.21.15.105:0/2123169465 wait complete. 2024-04-22T06:37:03.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:03.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- 172.21.15.105:0/4218396597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103b50 msgr2=0x7f537810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 --2- 172.21.15.105:0/4218396597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103b50 0x7f537810a3e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5370009ca0 tx=0x7f537002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- 172.21.15.105:0/4218396597 shutdown_connections 2024-04-22T06:37:03.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 --2- 172.21.15.105:0/4218396597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103b50 0x7f537810a3e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 --2- 172.21.15.105:0/4218396597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103190 0x7f5378103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 --2- 172.21.15.105:0/4218396597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 0x7f53781023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- 172.21.15.105:0/4218396597 >> 172.21.15.105:0/4218396597 conn(0x7f53780fd530 msgr2=0x7f53780ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- 172.21.15.105:0/4218396597 shutdown_connections 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- 172.21.15.105:0/4218396597 wait complete. 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 Processor -- start 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.372+0000 7f537e1c3640 1 -- start start 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 0x7f5378198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103b50 0x7f537819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f537810db40 con 0x7f5378103190 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f537810d9c0 con 0x7f5378101fa0 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f537810dcc0 con 0x7f5378103b50 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53048/0 (socket says 172.21.15.105:53048) 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 -- 172.21.15.105:0/3244758019 learned_addr learned my addr 172.21.15.105:0/3244758019 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:03.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5377fff640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103b50 0x7f537819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f53777fe640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 0x7f5378198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 -- 172.21.15.105:0/3244758019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103b50 msgr2=0x7f537819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103b50 0x7f537819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 -- 172.21.15.105:0/3244758019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 msgr2=0x7f5378198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 0x7f5378198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 -- 172.21.15.105:0/3244758019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537819fd20 con 0x7f5378103190 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5376ffd640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c920 tx=0x7f536c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c007c70 con 0x7f5378103190 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537819ffb0 con 0x7f5378103190 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f536c007e10 con 0x7f5378103190 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c018820 con 0x7f5378103190 2024-04-22T06:37:03.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.373+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53781a0520 con 0x7f5378103190 2024-04-22T06:37:03.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.374+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5378109260 con 0x7f5378103190 2024-04-22T06:37:03.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.375+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f536c00f450 con 0x7f5378103190 2024-04-22T06:37:03.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.376+0000 7f5374ff9640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 0x7f53600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.376+0000 7f53777fe640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 0x7f53600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.376+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f536c0d0580 con 0x7f5378103190 2024-04-22T06:37:03.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.376+0000 7f53777fe640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 0x7f53600a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5378102ff0 tx=0x7f5368008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.380+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f536c014030 con 0x7f5378103190 2024-04-22T06:37:03.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.596+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f537802d7c0 con 0x7f5378103190 2024-04-22T06:37:03.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.596+0000 7f5374ff9640 1 -- 172.21.15.105:0/3244758019 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7f536c09cf00 con 0x7f5378103190 2024-04-22T06:37:03.595 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:03.595 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:37:03.596 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:03.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 msgr2=0x7f53600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 0x7f53600a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5378102ff0 tx=0x7f5368008040 comp rx=0 tx=0).stop 2024-04-22T06:37:03.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 msgr2=0x7f5378198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c920 tx=0x7f536c00cde0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 shutdown_connections 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53600a3c40 0x7f53600a6100 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5378103b50 0x7f537819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5378103190 0x7f5378198640 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 --2- 172.21.15.105:0/3244758019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5378101fa0 0x7f5378198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.598+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 >> 172.21.15.105:0/3244758019 conn(0x7f53780fd530 msgr2=0x7f5378106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.599+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 shutdown_connections 2024-04-22T06:37:03.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.599+0000 7f537e1c3640 1 -- 172.21.15.105:0/3244758019 wait complete. 2024-04-22T06:37:03.609 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:03.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/3235820557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38107cd0 msgr2=0x7f8d3810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/3235820557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38107cd0 0x7f8d3810a0c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009ca0 tx=0x7f8d3002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/3235820557 shutdown_connections 2024-04-22T06:37:03.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/3235820557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38107cd0 0x7f8d3810a0c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/3235820557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 0x7f8d38107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/3235820557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38102a70 0x7f8d38104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/3235820557 >> 172.21.15.105:0/3235820557 conn(0x7f8d380fc3e0 msgr2=0x7f8d380fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/3235820557 shutdown_connections 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.708+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/3235820557 wait complete. 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 Processor -- start 2024-04-22T06:37:03.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 -- start start 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 0x7f8d381986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38107cd0 0x7f8d3819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3810dac0 con 0x7f8d38102a70 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3810d940 con 0x7f8d381053a0 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d3e6e4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3810dc40 con 0x7f8d38107cd0 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d37fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d37fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53072/0 (socket says 172.21.15.105:53072) 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.709+0000 7f8d37fff640 1 -- 172.21.15.105:0/580853343 learned_addr learned my addr 172.21.15.105:0/580853343 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d377fe640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 0x7f8d381986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d3cc81640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38107cd0 0x7f8d3819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 -- 172.21.15.105:0/580853343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38107cd0 msgr2=0x7f8d3819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38107cd0 0x7f8d3819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 -- 172.21.15.105:0/580853343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 msgr2=0x7f8d381986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 0x7f8d381986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 -- 172.21.15.105:0/580853343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3819fd90 con 0x7f8d38102a70 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d37fff640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800d770 tx=0x7f8d2800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28017070 con 0x7f8d38102a70 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d381a0080 con 0x7f8d38102a70 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d280054d0 con 0x7f8d38102a70 2024-04-22T06:37:03.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28004c30 con 0x7f8d38102a70 2024-04-22T06:37:03.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.710+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d381a05d0 con 0x7f8d38102a70 2024-04-22T06:37:03.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.711+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d380fded0 con 0x7f8d38102a70 2024-04-22T06:37:03.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.712+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8d28004e10 con 0x7f8d38102a70 2024-04-22T06:37:03.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.713+0000 7f8d357fa640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 0x7f8d200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:03.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.713+0000 7f8d377fe640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 0x7f8d200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:03.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.713+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f8d280d0000 con 0x7f8d38102a70 2024-04-22T06:37:03.713 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.713+0000 7f8d377fe640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 0x7f8d200a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c005fd0 tx=0x7f8d2c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:03.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.717+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d2809c980 con 0x7f8d38102a70 2024-04-22T06:37:03.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.895+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d38062b50 con 0x7f8d38102a70 2024-04-22T06:37:03.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.897+0000 7f8d357fa640 1 -- 172.21.15.105:0/580853343 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7f8d2809c280 con 0x7f8d38102a70 2024-04-22T06:37:03.896 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:03.896 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 msgr2=0x7f8d200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 0x7f8d200a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c005fd0 tx=0x7f8d2c005850 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 msgr2=0x7f8d38198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800d770 tx=0x7f8d2800dc30 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 shutdown_connections 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f8d200a3c40 0x7f8d200a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d38107cd0 0x7f8d3819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d381053a0 0x7f8d381986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 --2- 172.21.15.105:0/580853343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d38102a70 0x7f8d38198170 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 >> 172.21.15.105:0/580853343 conn(0x7f8d380fc3e0 msgr2=0x7f8d38103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 shutdown_connections 2024-04-22T06:37:03.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:03.899+0000 7f8d3e6e4640 1 -- 172.21.15.105:0/580853343 wait complete. 2024-04-22T06:37:03.907 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:03 smithi105 ceph-mon[27916]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:03.908 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:03 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2919073211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.908 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:03 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2123169465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.908 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:03 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3244758019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:03.909 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:03 smithi177 ceph-mon[31346]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:03 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2919073211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:03 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2123169465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:03.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:03 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3244758019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.008+0000 7f52b5bab640 1 -- 172.21.15.105:0/585589094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0102980 msgr2=0x7f52b01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.008+0000 7f52b5bab640 1 --2- 172.21.15.105:0/585589094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0102980 0x7f52b01092a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009ca0 tx=0x7f52a802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- 172.21.15.105:0/585589094 shutdown_connections 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- 172.21.15.105:0/585589094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0102980 0x7f52b01092a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- 172.21.15.105:0/585589094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0101fc0 0x7f52b0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- 172.21.15.105:0/585589094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 0x7f52b0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- 172.21.15.105:0/585589094 >> 172.21.15.105:0/585589094 conn(0x7f52b00fc3e0 msgr2=0x7f52b00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- 172.21.15.105:0/585589094 shutdown_connections 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- 172.21.15.105:0/585589094 wait complete. 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 Processor -- start 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- start start 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 0x7f52b0069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 0x7f52b019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010ca20 con 0x7f52b0101fc0 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010c8a0 con 0x7f52b0102980 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52b5bab640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010cba0 con 0x7f52b0100e70 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52a7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53084/0 (socket says 172.21.15.105:53084) 2024-04-22T06:37:04.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.009+0000 7f52a7fff640 1 -- 172.21.15.105:0/1099802977 learned_addr learned my addr 172.21.15.105:0/1099802977 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52affff640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 0x7f52b019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52af7fe640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 0x7f52b0069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 -- 172.21.15.105:0/1099802977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 msgr2=0x7f52b0069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 0x7f52b0069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 -- 172.21.15.105:0/1099802977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 msgr2=0x7f52b019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 0x7f52b019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 -- 172.21.15.105:0/1099802977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b019fd80 con 0x7f52b0101fc0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52affff640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 0x7f52b019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52a7fff640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f529800c970 tx=0x7f529800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298007c10 con 0x7f52b0101fc0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5298007db0 con 0x7f52b0101fc0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b019ffb0 con 0x7f52b0101fc0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529800f870 con 0x7f52b0101fc0 2024-04-22T06:37:04.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.010+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b01a04d0 con 0x7f52b0101fc0 2024-04-22T06:37:04.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.011+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b0108120 con 0x7f52b0101fc0 2024-04-22T06:37:04.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.012+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f529800fa10 con 0x7f52b0101fc0 2024-04-22T06:37:04.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.013+0000 7f52ad7fa640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 0x7f529c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.013+0000 7f52af7fe640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 0x7f529c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.013+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f52980d0c10 con 0x7f52b0101fc0 2024-04-22T06:37:04.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.013+0000 7f52af7fe640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 0x7f529c0a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f52a000a810 tx=0x7f52a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.017+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f529809d5b0 con 0x7f52b0101fc0 2024-04-22T06:37:04.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:03 smithi112 ceph-mon[31236]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:04.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:03 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2919073211' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:04.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:03 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2123169465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:04.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:03 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3244758019' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:04.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.232+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52b002d5c0 con 0x7f52b0101fc0 2024-04-22T06:37:04.230 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.232+0000 7f52ad7fa640 1 -- 172.21.15.105:0/1099802977 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7f5298017e80 con 0x7f52b0101fc0 2024-04-22T06:37:04.231 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:04.231 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:37:04.231 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 msgr2=0x7f529c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 0x7f529c0a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f52a000a810 tx=0x7f52a0008040 comp rx=0 tx=0).stop 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 msgr2=0x7f52b019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f529800c970 tx=0x7f529800ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 shutdown_connections 2024-04-22T06:37:04.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f529c0a3c40 0x7f529c0a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52b0102980 0x7f52b019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52b0101fc0 0x7f52b019ce50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 --2- 172.21.15.105:0/1099802977 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52b0100e70 0x7f52b0069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 >> 172.21.15.105:0/1099802977 conn(0x7f52b00fc3e0 msgr2=0x7f52b01052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 shutdown_connections 2024-04-22T06:37:04.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.234+0000 7f52b5bab640 1 -- 172.21.15.105:0/1099802977 wait complete. 2024-04-22T06:37:04.244 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:35:40.134334+0000', 'modified': '2024-04-22T06:35:47.003690+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24472, 'mds_2': 24461, 'mds_3': 24454, 'mds_4': 14640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6839/256609249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 256609249}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 256609249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14640': {'gid': 14640, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6841/2147123680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6840', 'nonce': 2147123680}, {'type': 'v1', 'addr': '172.21.15.105:6841', 'nonce': 2147123680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24397': {'gid': 24397, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6833/624977252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 624977252}, {'type': 'v1', 'addr': '172.21.15.177:6833', 'nonce': 624977252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/583171810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 583171810}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 583171810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/3883368202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 3883368202}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 3883368202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/3449982583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 3449982583}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 3449982583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.177:6837/4114728122', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 4114728122}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 4114728122}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6837/3970644798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3970644798}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3970644798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.177:6839/1381266768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 1381266768}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 1381266768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/4112363218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 4112363218}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 4112363218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24472, 24500, 24461, 24454, 14640]} 2024-04-22T06:37:04.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T06:37:04.245 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 -- 172.21.15.105:0/3918640687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc101fe0 msgr2=0x7f59bc102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 --2- 172.21.15.105:0/3918640687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc101fe0 0x7f59bc102460 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009900 tx=0x7f59ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 -- 172.21.15.105:0/3918640687 shutdown_connections 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 --2- 172.21.15.105:0/3918640687 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc1029a0 0x7f59bc1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 --2- 172.21.15.105:0/3918640687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc101fe0 0x7f59bc102460 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 --2- 172.21.15.105:0/3918640687 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 -- 172.21.15.105:0/3918640687 >> 172.21.15.105:0/3918640687 conn(0x7f59bc0fc440 msgr2=0x7f59bc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 -- 172.21.15.105:0/3918640687 shutdown_connections 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 -- 172.21.15.105:0/3918640687 wait complete. 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.345+0000 7f59c20fd640 1 Processor -- start 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 -- start start 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 0x7f59bc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc10ca30 con 0x7f59bc1029a0 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc10c8b0 con 0x7f59bc100e90 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59c20fd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc10cbb0 con 0x7f59bc101fe0 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53102/0 (socket says 172.21.15.105:53102) 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 -- 172.21.15.105:0/2815669186 learned_addr learned my addr 172.21.15.105:0/2815669186 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59baffd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 0x7f59bc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 -- 172.21.15.105:0/2815669186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 msgr2=0x7f59bc198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 0x7f59bc198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 -- 172.21.15.105:0/2815669186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 msgr2=0x7f59bc198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 -- 172.21.15.105:0/2815669186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59bc19fe70 con 0x7f59bc1029a0 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59baffd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 0x7f59bc198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59bbfff640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f59b400cad0 tx=0x7f59b400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.346+0000 7f59b8ff9640 1 -- 172.21.15.105:0/2815669186 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b400ae20 con 0x7f59bc1029a0 2024-04-22T06:37:04.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.347+0000 7f59b8ff9640 1 -- 172.21.15.105:0/2815669186 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b4005020 con 0x7f59bc1029a0 2024-04-22T06:37:04.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.347+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59bc1a0160 con 0x7f59bc1029a0 2024-04-22T06:37:04.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.347+0000 7f59b8ff9640 1 -- 172.21.15.105:0/2815669186 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b400f6b0 con 0x7f59bc1029a0 2024-04-22T06:37:04.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.347+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59bc1a0680 con 0x7f59bc1029a0 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59b8ff9640 1 -- 172.21.15.105:0/2815669186 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f59b40051d0 con 0x7f59bc1029a0 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 Processor -- start 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 -- start start 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x561607152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x561607152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5616070b6b10 con 0x5616071532d0 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561607054240 con 0x561607151e40 2024-04-22T06:37:04.356 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5616070b2ad0 con 0x561607152830 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53118/0 (socket says 172.21.15.105:53118) 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 -- 172.21.15.105:0/1765247991 learned_addr learned my addr 172.21.15.105:0/1765247991 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59b8ff9640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 0x7f59a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x561607152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x561607152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59bbfff640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 0x7f59a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 -- 172.21.15.105:0/1765247991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 msgr2=0x561607152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x561607152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 -- 172.21.15.105:0/1765247991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 msgr2=0x561607152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x561607152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.349+0000 7f59baffd640 1 -- 172.21.15.105:0/1765247991 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561606dc92d0 con 0x5616071532d0 2024-04-22T06:37:04.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59b8ff9640 1 -- 172.21.15.105:0/2815669186 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f59b40d0550 con 0x7f59bc1029a0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59baffd640 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f59ac005ec0 tx=0x7f59ac005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ad3994f08c3cd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59a1ffb640 1 -- 172.21.15.105:0/1765247991 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac004340 con 0x5616071532d0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59bbfff640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 0x7f59a40a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f59b4098c30 tx=0x7f59b4087040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59a1ffb640 1 -- 172.21.15.105:0/1765247991 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59a1ffb640 1 -- 172.21.15.105:0/1765247991 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac004c00 con 0x5616071532d0 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1765247991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 msgr2=0x561607159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.350+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f59ac005ec0 tx=0x7f59ac005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1765247991 shutdown_connections 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607159eb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x561607152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1765247991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x561607152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1765247991 >> 172.21.15.105:0/1765247991 conn(0x56160714df40 msgr2=0x56160714c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1765247991 shutdown_connections 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1765247991 wait complete. 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- start start 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 Processor -- start 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x56160715c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56160715a450 con 0x5616071532d0 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56160703f190 con 0x561607152830 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56160715a5d0 con 0x561607151e40 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59c4dc7740 10 client.0 _ll_register_callbacks cb 0x56160703a8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53124/0 (socket says 172.21.15.105:53124) 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.351+0000 7f59baffd640 1 -- 172.21.15.105:0/939818076 learned_addr learned my addr 172.21.15.105:0/939818076 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 -- 172.21.15.105:0/939818076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 msgr2=0x56160715bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 -- 172.21.15.105:0/939818076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 msgr2=0x56160715c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x56160715c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 -- 172.21.15.105:0/939818076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561606e936b0 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59baffd640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c840 tx=0x7f59ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7819202b12bb10dd server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac000e60 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f59ac005040 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59ac005240 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 10 client.15057 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561607166560 con 0x5616071532d0 2024-04-22T06:37:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f59ac003d10 con 0x5616071532d0 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 10 client.15057 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 10 client.15057 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.352+0000 7f59c4dc7740 20 client.15057 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x561607168750 0x561607169bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f59c4dc7740 4 client.15057 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616070c4530 con 0x561607168750 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x561607168750 0x561607169bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x561607168750 0x561607169bc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=15c066ab3e27c80a server_cookie=bd7d0fb64533bdd0 in_seq=0 out_seq=0 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f599affd640 10 client.15057 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.353+0000 7f599affd640 1 -- 172.21.15.105:0/939818076 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616070c4530 con 0x561607168750 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f599affd640 10 client.15057 handle_command_reply: tid=0 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 client.15057 shutdown 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 msgr2=0x561607163210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c840 tx=0x7f59ac02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 shutdown_connections 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607163210 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x56160715c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x561607168750 0x561607169bc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/939818076 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f599affd640 0 client.15057 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 >> 172.21.15.105:0/939818076 conn(0x56160714df40 msgr2=0x56160714ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 shutdown_connections 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 1 -- 172.21.15.105:0/939818076 wait complete. 2024-04-22T06:37:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.354+0000 7f59c4dc7740 20 client.15057 trim_cache size 0 max 16384 2024-04-22T06:37:04.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 Processor -- start 2024-04-22T06:37:04.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 -- start start 2024-04-22T06:37:04.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x5616071678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x5616071631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561607167f00 con 0x5616071532d0 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5616071685d0 con 0x561607151e40 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x561607168ca0 con 0x561607152830 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53138/0 (socket says 172.21.15.105:53138) 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 -- 172.21.15.105:0/1260710967 learned_addr learned my addr 172.21.15.105:0/1260710967 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 -- 172.21.15.105:0/1260710967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 msgr2=0x5616071631d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x5616071631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 -- 172.21.15.105:0/1260710967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 msgr2=0x5616071678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x5616071678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 -- 172.21.15.105:0/1260710967 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56160714dd40 con 0x5616071532d0 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.372+0000 7f59baffd640 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c170 tx=0x7f59ac005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce64d003bf994fb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59a17fa640 1 -- 172.21.15.105:0/1260710967 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac065070 con 0x5616071532d0 2024-04-22T06:37:04.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59a17fa640 1 -- 172.21.15.105:0/1260710967 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59a17fa640 1 -- 172.21.15.105:0/1260710967 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac043840 con 0x5616071532d0 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1260710967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 msgr2=0x56160715b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59ac02c170 tx=0x7f59ac005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1260710967 shutdown_connections 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x56160715b0b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607152830 0x5616071631d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/1260710967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607151e40 0x5616071678b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1260710967 >> 172.21.15.105:0/1260710967 conn(0x56160714df40 msgr2=0x56160715a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1260710967 shutdown_connections 2024-04-22T06:37:04.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- 172.21.15.105:0/1260710967 wait complete. 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.373+0000 7f59c4dc7740 1 -- start start 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 Processor -- start 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x561607161700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x561607169a40 con 0x5616071532d0 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x561607169370 con 0x561607152830 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56160716a110 con 0x561607151e40 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59c4dc7740 10 client.0 _ll_register_callbacks cb 0x7f59ac02cc80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53154/0 (socket says 172.21.15.105:53154) 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 -- 172.21.15.105:0/4088027738 learned_addr learned my addr 172.21.15.105:0/4088027738 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 -- 172.21.15.105:0/4088027738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 msgr2=0x56160715f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x561607161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 -- 172.21.15.105:0/4088027738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 msgr2=0x561607161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x561607161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 -- 172.21.15.105:0/4088027738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561607162440 con 0x5616071532d0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f59baffd640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0059d0 tx=0x7f59ac02cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934ddea16d7161d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.374+0000 7f599affd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac065070 con 0x5616071532d0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f59ac0043d0 con 0x5616071532d0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f59ac043b60 con 0x5616071532d0 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 10 client.15063 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5616071ce700 con 0x5616071532d0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59ac048070 con 0x5616071532d0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f59ac069020 con 0x5616071532d0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 10 client.15063 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 10 client.15063 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 20 client.15063 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x5616071d2f20 0x5616071d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 4 client.15063 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "status"}) -- 0x5616070be3a0 con 0x5616071d2f20 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x5616071d2f20 0x5616071d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f59bb7fe640 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x5616071d2f20 0x5616071d1c30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ea761b38361548d2 server_cookie=a312d78c47c1a6ee in_seq=0 out_seq=0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.375+0000 7f599affd640 10 client.15063 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f599affd640 1 -- 172.21.15.105:0/4088027738 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5616070be3a0 con 0x5616071d2f20 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f599affd640 10 client.15063 handle_command_reply: tid=0 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 client.15063 shutdown 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 msgr2=0x561607161c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0059d0 tx=0x7f59ac02cdd0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 shutdown_connections 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5616071532d0 0x561607161c40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x561607152830 0x561607161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x5616071d2f20 0x5616071d1c30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 --2- 172.21.15.105:0/4088027738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x561607151e40 0x56160715f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 >> 172.21.15.105:0/4088027738 conn(0x56160714df40 msgr2=0x56160715b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f599affd640 0 client.15063 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 shutdown_connections 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 1 -- 172.21.15.105:0/4088027738 wait complete. 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c4dc7740 20 client.15063 trim_cache size 0 max 16384 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:04.384 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 4, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 14640, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 78.377046358000001, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 91.967013944000001 2024-04-22T06:37:04.385 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 msgr2=0x7f59a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 0x7f59a40a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f59b4098c30 tx=0x7f59b4087040 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 msgr2=0x7f59bc19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f59b400cad0 tx=0x7f59b400cf90 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 shutdown_connections 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f59a40a3c40 0x7f59a40a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59bc1029a0 0x7f59bc19f800 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59bc101fe0 0x7f59bc198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.376+0000 7f59c20fd640 1 --2- 172.21.15.105:0/2815669186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59bc100e90 0x7f59bc198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.377+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 >> 172.21.15.105:0/2815669186 conn(0x7f59bc0fc440 msgr2=0x7f59bc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.377+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 shutdown_connections 2024-04-22T06:37:04.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.377+0000 7f59c20fd640 1 -- 172.21.15.105:0/2815669186 wait complete. 2024-04-22T06:37:04.388 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 -- 172.21.15.105:0/2755627308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 msgr2=0x7f29e4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/2755627308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4104e80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009900 tx=0x7f29d402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 -- 172.21.15.105:0/2755627308 shutdown_connections 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/2755627308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29e4107cf0 0x7f29e410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/2755627308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e41053c0 0x7f29e41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/2755627308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4104e80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 -- 172.21.15.105:0/2755627308 >> 172.21.15.105:0/2755627308 conn(0x7f29e40fc440 msgr2=0x7f29e40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 -- 172.21.15.105:0/2755627308 shutdown_connections 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.488+0000 7f29ea6a8640 1 -- 172.21.15.105:0/2755627308 wait complete. 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 Processor -- start 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 -- start start 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29e41053c0 0x7f29e41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e4107cf0 0x7f29e419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410db50 con 0x7f29e4102a90 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410d9d0 con 0x7f29e4107cf0 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29ea6a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f29e410dcd0 con 0x7f29e41053c0 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53168/0 (socket says 172.21.15.105:53168) 2024-04-22T06:37:04.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 -- 172.21.15.105:0/1238437135 learned_addr learned my addr 172.21.15.105:0/1238437135 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 -- 172.21.15.105:0/1238437135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29e41053c0 msgr2=0x7f29e41987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29e41053c0 0x7f29e41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e4107cf0 0x7f29e419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 -- 172.21.15.105:0/1238437135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e4107cf0 msgr2=0x7f29e419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e4107cf0 0x7f29e419f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.489+0000 7f29e3fff640 1 -- 172.21.15.105:0/1238437135 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e419fe80 con 0x7f29e4102a90 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036040 tx=0x7f29d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29e17fa640 1 -- 172.21.15.105:0/1238437135 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4043020 con 0x7f29e4102a90 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29e17fa640 1 -- 172.21.15.105:0/1238437135 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29d40054c0 con 0x7f29e4102a90 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29e17fa640 1 -- 172.21.15.105:0/1238437135 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4003db0 con 0x7f29e4102a90 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e41a0110 con 0x7f29e4102a90 2024-04-22T06:37:04.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.490+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e41a06e0 con 0x7f29e4102a90 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29e17fa640 1 -- 172.21.15.105:0/1238437135 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f29d4046020 con 0x7f29e4102a90 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 Processor -- start 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 -- start start 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29e17fa640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 0x7f29cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54ce40 0x56031c54d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c554eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56031c4b1b10 con 0x56031c54d830 2024-04-22T06:37:04.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56031c44f240 con 0x56031c54e2d0 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29ed379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56031c4adad0 con 0x56031c54ce40 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.492+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 0x7f29cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e17fa640 1 -- 172.21.15.105:0/1238437135 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f29d40f2920 con 0x7f29e4102a90 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53180/0 (socket says 172.21.15.105:53180) 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 -- 172.21.15.105:0/2302112803 learned_addr learned my addr 172.21.15.105:0/2302112803 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e8c45640 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c554eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e3fff640 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54ce40 0x56031c54d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 -- 172.21.15.105:0/2302112803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54ce40 msgr2=0x56031c54d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54ce40 0x56031c54d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 -- 172.21.15.105:0/2302112803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 msgr2=0x56031c554eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c554eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 -- 172.21.15.105:0/2302112803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56031c1c42d0 con 0x56031c54d830 2024-04-22T06:37:04.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 0x7f29cc0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29dc008e40 tx=0x7f29dc011610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29e37fe640 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f29d800a970 tx=0x7f29d800ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862459d06079bb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29ca7fc640 1 -- 172.21.15.105:0/2302112803 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d800cd00 con 0x56031c54d830 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29ca7fc640 1 -- 172.21.15.105:0/2302112803 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29d80035f0 con 0x56031c54d830 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.493+0000 7f29ca7fc640 1 -- 172.21.15.105:0/2302112803 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d800f6b0 con 0x56031c54d830 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- 172.21.15.105:0/2302112803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 msgr2=0x56031c54dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f29d800a970 tx=0x7f29d800ae30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- 172.21.15.105:0/2302112803 shutdown_connections 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c554eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54d830 0x56031c54dcb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 --2- 172.21.15.105:0/2302112803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54ce40 0x56031c54d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- 172.21.15.105:0/2302112803 >> 172.21.15.105:0/2302112803 conn(0x56031c548f40 msgr2=0x56031c547490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- 172.21.15.105:0/2302112803 shutdown_connections 2024-04-22T06:37:04.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- 172.21.15.105:0/2302112803 wait complete. 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 -- start start 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.494+0000 7f29ed379740 1 Processor -- start 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c556e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c5573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56031c5554b0 con 0x56031c54e2d0 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56031c43a190 con 0x56031c54ce40 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56031c555630 con 0x56031c54d830 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 10 client.0 _ll_register_callbacks cb 0x56031c4358b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53184/0 (socket says 172.21.15.105:53184) 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 -- 172.21.15.105:0/3664668232 learned_addr learned my addr 172.21.15.105:0/3664668232 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e37fe640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c5573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e3fff640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c556e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 -- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 msgr2=0x56031c5573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c5573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 -- 172.21.15.105:0/3664668232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 msgr2=0x56031c556e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c556e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 -- 172.21.15.105:0/3664668232 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56031c28e6b0 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29e8c45640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f29dc017e00 tx=0x7f29dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c25af3e6b940f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc024020 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f29dc020650 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29d80035f0 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.495+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f29dc017e30 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 10 client.15075 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56031c561830 con 0x56031c54e2d0 2024-04-22T06:37:04.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29d80035f0 con 0x56031c54e2d0 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f29dc01f870 con 0x56031c54e2d0 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 10 client.15075 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 10 client.15075 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 20 client.15075 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c563f00 0x56031c565370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 4 client.15075 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56031c4bf530 con 0x56031c563f00 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.496+0000 7f29e3fff640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c563f00 0x56031c565370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.497+0000 7f29e3fff640 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c563f00 0x56031c565370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91de2b795a7a1eca server_cookie=613a529ef324a94a in_seq=0 out_seq=0 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.497+0000 7f29c37fe640 10 client.15075 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.497+0000 7f29c37fe640 1 -- 172.21.15.105:0/3664668232 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56031c4bf530 con 0x56031c563f00 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.497+0000 7f29c37fe640 10 client.15075 handle_command_reply: tid=0 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 client.15075 shutdown 2024-04-22T06:37:04.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 msgr2=0x56031c55e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f29dc017e00 tx=0x7f29dc004a40 comp rx=0 tx=0).stop 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 shutdown_connections 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c563f00 0x56031c565370 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e400 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c5573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29c37fe640 0 client.15075 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 --2- 172.21.15.105:0/3664668232 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c556e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 >> 172.21.15.105:0/3664668232 conn(0x56031c548f40 msgr2=0x56031c546a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 shutdown_connections 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 1 -- 172.21.15.105:0/3664668232 wait complete. 2024-04-22T06:37:04.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.498+0000 7f29ed379740 20 client.15075 trim_cache size 0 max 16384 2024-04-22T06:37:04.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 Processor -- start 2024-04-22T06:37:04.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 -- start start 2024-04-22T06:37:04.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c5517b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c551cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56031c5636b0 con 0x56031c54e2d0 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56031c563d80 con 0x56031c54ce40 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29ed379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56031c564450 con 0x56031c54d830 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53986/0 (socket says 172.21.15.105:53986) 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 -- 172.21.15.105:0/1861018430 learned_addr learned my addr 172.21.15.105:0/1861018430 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 -- 172.21.15.105:0/1861018430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 msgr2=0x56031c551cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c551cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 -- 172.21.15.105:0/1861018430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 msgr2=0x56031c5517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c5517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 -- 172.21.15.105:0/1861018430 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56031c5634d0 con 0x56031c54e2d0 2024-04-22T06:37:04.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.516+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29dc000f80 tx=0x7f29dc008ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc97eed28324331c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29c9ffb640 1 -- 172.21.15.105:0/1861018430 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc024070 con 0x56031c54e2d0 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29c9ffb640 1 -- 172.21.15.105:0/1861018430 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29dc01b050 con 0x56031c54e2d0 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29c9ffb640 1 -- 172.21.15.105:0/1861018430 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc020b10 con 0x56031c54e2d0 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- 172.21.15.105:0/1861018430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 msgr2=0x56031c55e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f29dc000f80 tx=0x7f29dc008ac0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- 172.21.15.105:0/1861018430 shutdown_connections 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54e2d0 0x56031c55e3c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c551cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 --2- 172.21.15.105:0/1861018430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54ce40 0x56031c5517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- 172.21.15.105:0/1861018430 >> 172.21.15.105:0/1861018430 conn(0x56031c548f40 msgr2=0x56031c556e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- 172.21.15.105:0/1861018430 shutdown_connections 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- 172.21.15.105:0/1861018430 wait complete. 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.517+0000 7f29ed379740 1 -- start start 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 Processor -- start 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c55a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c55d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56031c5651f0 con 0x56031c54ce40 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56031c564b20 con 0x56031c54e2d0 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56031c5658c0 con 0x56031c54d830 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29ed379740 10 client.0 _ll_register_callbacks cb 0x7f29dc008040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54000/0 (socket says 172.21.15.105:54000) 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 -- 172.21.15.105:0/1787931290 learned_addr learned my addr 172.21.15.105:0/1787931290 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e37fe640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c55a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e8c45640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c55d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 -- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 msgr2=0x56031c55a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c55a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 -- 172.21.15.105:0/1787931290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 msgr2=0x56031c55d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c55d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 -- 172.21.15.105:0/1787931290 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56031c55d9c0 con 0x56031c54ce40 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e37fe640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c55a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29e3fff640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f29d40bb680 tx=0x7f29d4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cdd316f8862ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.518+0000 7f29c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d40becb0 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f29d40bb320 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29dc01b050 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f29d40c3060 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 10 client.15081 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56031c5ca270 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29dc01b050 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f29d40c7490 con 0x56031c54ce40 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 10 client.15081 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 10 client.15081 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 20 client.15081 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c5cee90 0x56031c5cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 4 client.15081 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "status"}) -- 0x56031c4b93a0 con 0x56031c5cee90 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.519+0000 7f29e37fe640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c5cee90 0x56031c5cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29e37fe640 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c5cee90 0x56031c5cdba0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b386add3cfe9a1a server_cookie=d4827918b1b236b3 in_seq=0 out_seq=0 2024-04-22T06:37:04.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29c37fe640 10 client.15081 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29c37fe640 1 -- 172.21.15.105:0/1787931290 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56031c4b93a0 con 0x56031c5cee90 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29c37fe640 10 client.15081 handle_command_reply: tid=0 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 client.15081 shutdown 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 msgr2=0x56031c559e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f29d40bb680 tx=0x7f29d4009260 comp rx=0 tx=0).stop 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 shutdown_connections 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56031c5cee90 0x56031c5cdba0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56031c54e2d0 0x56031c55d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56031c54d830 0x56031c55a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29c37fe640 0 client.15081 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 --2- 172.21.15.105:0/1787931290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56031c54ce40 0x56031c559e70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 >> 172.21.15.105:0/1787931290 conn(0x56031c548f40 msgr2=0x56031c556e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 shutdown_connections 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.520+0000 7f29ed379740 1 -- 172.21.15.105:0/1787931290 wait complete. 2024-04-22T06:37:04.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ed379740 20 client.15081 trim_cache size 0 max 16384 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 3, 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24454, 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 79.484859950000001, 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:04.527 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:04.528 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 103.004219083 2024-04-22T06:37:04.528 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:04.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 msgr2=0x7f29cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 0x7f29cc0a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29dc008e40 tx=0x7f29dc011610 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 msgr2=0x7f29e4198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036040 tx=0x7f29d40042f0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 shutdown_connections 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f29cc0a3c40 0x7f29cc0a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29e4107cf0 0x7f29e419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29e41053c0 0x7f29e41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 --2- 172.21.15.105:0/1238437135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29e4102a90 0x7f29e4198260 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 >> 172.21.15.105:0/1238437135 conn(0x7f29e40fc440 msgr2=0x7f29e4103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 shutdown_connections 2024-04-22T06:37:04.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.521+0000 7f29ea6a8640 1 -- 172.21.15.105:0/1238437135 wait complete. 2024-04-22T06:37:04.531 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 -- 172.21.15.105:0/4048464194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34102ad0 msgr2=0x7f3f34104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 --2- 172.21.15.105:0/4048464194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34102ad0 0x7f3f34104ec0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009900 tx=0x7f3f2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 -- 172.21.15.105:0/4048464194 shutdown_connections 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 --2- 172.21.15.105:0/4048464194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34107d30 0x7f3f3410a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 --2- 172.21.15.105:0/4048464194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 0x7f3f341077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 --2- 172.21.15.105:0/4048464194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34102ad0 0x7f3f34104ec0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 -- 172.21.15.105:0/4048464194 >> 172.21.15.105:0/4048464194 conn(0x7f3f340fc420 msgr2=0x7f3f340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 -- 172.21.15.105:0/4048464194 shutdown_connections 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.631+0000 7f3f39723640 1 -- 172.21.15.105:0/4048464194 wait complete. 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 Processor -- start 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 -- start start 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34102ad0 0x7f3f34102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 0x7f3f34100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410db10 con 0x7f3f34107d30 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410d990 con 0x7f3f34102ad0 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f39723640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410dc90 con 0x7f3f34105400 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54028/0 (socket says 172.21.15.105:54028) 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 -- 172.21.15.105:0/3034798659 learned_addr learned my addr 172.21.15.105:0/3034798659 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34102ad0 0x7f3f34102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f327fc640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 0x7f3f34100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 -- 172.21.15.105:0/3034798659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 msgr2=0x7f3f34100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 0x7f3f34100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 -- 172.21.15.105:0/3034798659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34102ad0 msgr2=0x7f3f34102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34102ad0 0x7f3f34102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.632+0000 7f3f337fe640 1 -- 172.21.15.105:0/3034798659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f34101850 con 0x7f3f34107d30 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f337fe640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00eb10 tx=0x7f3f2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f1bfff640 1 -- 172.21.15.105:0/3034798659 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c004b20 con 0x7f3f34107d30 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f341a23b0 con 0x7f3f34107d30 2024-04-22T06:37:04.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f1bfff640 1 -- 172.21.15.105:0/3034798659 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f2c004cc0 con 0x7f3f34107d30 2024-04-22T06:37:04.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f1bfff640 1 -- 172.21.15.105:0/3034798659 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c010670 con 0x7f3f34107d30 2024-04-22T06:37:04.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.633+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f341a28b0 con 0x7f3f34107d30 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f1bfff640 1 -- 172.21.15.105:0/3034798659 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3f2c005020 con 0x7f3f34107d30 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 Processor -- start 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 -- start start 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 0x5577adc89260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc90eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5577adbedb10 con 0x5577adc89830 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5577adb8b240 con 0x5577adc8a2d0 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5577adbe9ad0 con 0x5577adc88e40 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f32ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f32ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54044/0 (socket says 172.21.15.105:54044) 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f32ffd640 1 -- 172.21.15.105:0/1490072656 learned_addr learned my addr 172.21.15.105:0/1490072656 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.635+0000 7f3f1bfff640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 0x7f3f1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc90eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f337fe640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 0x7f3f1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 0x5577adc89260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 -- 172.21.15.105:0/1490072656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 msgr2=0x5577adc89260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 0x5577adc89260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 -- 172.21.15.105:0/1490072656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 msgr2=0x5577adc90eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc90eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 -- 172.21.15.105:0/1490072656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577ad9002d0 con 0x5577adc89830 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 0x5577adc89260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f1bfff640 1 -- 172.21.15.105:0/3034798659 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f3f2c0d0b30 con 0x7f3f34107d30 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009900 tx=0x7f3f24009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45e12eea6e540d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f337fe640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 0x7f3f1c0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c06f8c0 tx=0x7f3f2c06f8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f197fa640 1 -- 172.21.15.105:0/1490072656 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24045020 con 0x5577adc89830 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f197fa640 1 -- 172.21.15.105:0/1490072656 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f2402c9a0 con 0x5577adc89830 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.636+0000 7f3f197fa640 1 -- 172.21.15.105:0/1490072656 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240046c0 con 0x5577adc89830 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1490072656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 msgr2=0x5577adc89cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009900 tx=0x7f3f24009610 comp rx=0 tx=0).stop 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1490072656 shutdown_connections 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc90eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc89830 0x5577adc89cb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1490072656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc88e40 0x5577adc89260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1490072656 >> 172.21.15.105:0/1490072656 conn(0x5577adc84f40 msgr2=0x5577adc83490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1490072656 shutdown_connections 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1490072656 wait complete. 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- start start 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 Processor -- start 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc9a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.637+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5577adc91450 con 0x5577adc88e40 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5577adb76190 con 0x5577adc8a2d0 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5577adc915d0 con 0x5577adc89830 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 10 client.0 _ll_register_callbacks cb 0x5577adb718b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54056/0 (socket says 172.21.15.105:54056) 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 -- 172.21.15.105:0/1760245680 learned_addr learned my addr 172.21.15.105:0/1760245680 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 -- 172.21.15.105:0/1760245680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 msgr2=0x5577adc933a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 -- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 msgr2=0x5577adc9a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc9a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 -- 172.21.15.105:0/1760245680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577ad9ca6b0 con 0x5577adc88e40 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800efd0 tx=0x7f3f28011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3a966a23e858e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800fb40 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f3f2800fce0 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f2402c9a0 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f28005020 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 10 client.15093 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5577adc9d790 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.638+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f2402c9a0 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3f28025400 con 0x5577adc88e40 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 10 client.15093 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 20 client.15093 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577adc9fe80 0x5577adca12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 4 client.15093 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:04.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577adbfb530 con 0x5577adc9fe80 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577adc9fe80 0x5577adca12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f16ff5640 10 client.15093 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.639+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577adc9fe80 0x5577adca12f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17441ece9575701 server_cookie=6de35fad9db1299d in_seq=0 out_seq=0 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f16ff5640 1 -- 172.21.15.105:0/1760245680 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5577adbfb530 con 0x5577adc9fe80 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f16ff5640 10 client.15093 handle_command_reply: tid=0 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 client.15093 shutdown 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 msgr2=0x5577adc92e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800efd0 tx=0x7f3f28011bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 shutdown_connections 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc9a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577adc9fe80 0x5577adca12f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc933a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1760245680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc92e60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f16ff5640 0 client.15093 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:04.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 >> 172.21.15.105:0/1760245680 conn(0x5577adc84f40 msgr2=0x5577adc82a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 shutdown_connections 2024-04-22T06:37:04.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.640+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1760245680 wait complete. 2024-04-22T06:37:04.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.641+0000 7f3f3c3eb740 20 client.15093 trim_cache size 0 max 16384 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 Processor -- start 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 -- start start 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc9a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc83400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5577adc9f630 con 0x5577adc88e40 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5577adc9fd00 con 0x5577adc8a2d0 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5577adca03d0 con 0x5577adc89830 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54072/0 (socket says 172.21.15.105:54072) 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.658+0000 7f3f327fc640 1 -- 172.21.15.105:0/1910354433 learned_addr learned my addr 172.21.15.105:0/1910354433 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 -- 172.21.15.105:0/1910354433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 msgr2=0x5577adc9a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc9a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 -- 172.21.15.105:0/1910354433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 msgr2=0x5577adc83400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc83400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 -- 172.21.15.105:0/1910354433 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577adc84d40 con 0x5577adc88e40 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f327fc640 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3f28026f00 tx=0x7f3f28026f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244be13e52327e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f18ff9640 1 -- 172.21.15.105:0/1910354433 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28025600 con 0x5577adc88e40 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f18ff9640 1 -- 172.21.15.105:0/1910354433 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f280051c0 con 0x5577adc88e40 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f18ff9640 1 -- 172.21.15.105:0/1910354433 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800fcb0 con 0x5577adc88e40 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1910354433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 msgr2=0x5577adc89c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3f28026f00 tx=0x7f3f28026f30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1910354433 shutdown_connections 2024-04-22T06:37:04.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc8a2d0 0x5577adc83400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.659+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc9a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/1910354433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc88e40 0x5577adc89c10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1910354433 >> 172.21.15.105:0/1910354433 conn(0x5577adc84f40 msgr2=0x5577adc8d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1910354433 shutdown_connections 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/1910354433 wait complete. 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- start start 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 Processor -- start 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc88e40 0x5577adc96380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5577adca1170 con 0x5577adc8a2d0 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5577adca0aa0 con 0x5577adc88e40 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5577adca1840 con 0x5577adc89830 2024-04-22T06:37:04.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f3c3eb740 10 client.0 _ll_register_callbacks cb 0x7f3f2800e4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54076/0 (socket says 172.21.15.105:54076) 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 -- 172.21.15.105:0/3049058583 learned_addr learned my addr 172.21.15.105:0/3049058583 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc88e40 0x5577adc96380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 -- 172.21.15.105:0/3049058583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 msgr2=0x5577adc988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 -- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc88e40 msgr2=0x5577adc96380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc88e40 0x5577adc96380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f327fc640 1 -- 172.21.15.105:0/3049058583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577adc99730 con 0x5577adc8a2d0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.660+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f327fc640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800efd0 tx=0x7f3f280116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c842a12ffbd9a0c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28005370 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f3f28005510 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f280051c0 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3f28025a90 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 10 client.15099 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5577add05f20 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f280051c0 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3f2802d030 con 0x5577adc8a2d0 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 10 client.15099 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 10 client.15099 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 20 client.15099 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577add0a640 0x5577add07150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 4 client.15099 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:04.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "status"}) -- 0x5577adbf53a0 con 0x5577add0a640 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.661+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577add0a640 0x5577add07150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f16ff5640 10 client.15099 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f32ffd640 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577add0a640 0x5577add07150 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=995b657f195dbc8f server_cookie=23de936a43c682bb in_seq=0 out_seq=0 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f16ff5640 1 -- 172.21.15.105:0/3049058583 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5577adbf53a0 con 0x5577add0a640 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f16ff5640 10 client.15099 handle_command_reply: tid=0 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 client.15099 shutdown 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 msgr2=0x5577adc98eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800efd0 tx=0x7f3f280116c0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 shutdown_connections 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5577adc8a2d0 0x5577adc98eb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5577add0a640 0x5577add07150 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5577adc89830 0x5577adc988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f16ff5640 0 client.15099 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 --2- 172.21.15.105:0/3049058583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5577adc88e40 0x5577adc96380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 >> 172.21.15.105:0/3049058583 conn(0x5577adc84f40 msgr2=0x5577adc92a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 shutdown_connections 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 1 -- 172.21.15.105:0/3049058583 wait complete. 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.662+0000 7f3f3c3eb740 20 client.15099 trim_cache size 0 max 16384 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 2, 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24461, 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:04.669 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 80.637907244000004, 2024-04-22T06:37:04.670 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:04.670 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:04.670 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:04.670 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 100.41537411500001 2024-04-22T06:37:04.670 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 msgr2=0x7f3f1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 0x7f3f1c0a6100 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c06f8c0 tx=0x7f3f2c06f8f0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 msgr2=0x7f3f34100fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00eb10 tx=0x7f3f2c00c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 shutdown_connections 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3f1c0a3c40 0x7f3f1c0a6100 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3f34107d30 0x7f3f34100fc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3f34105400 0x7f3f34100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 --2- 172.21.15.105:0/3034798659 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3f34102ad0 0x7f3f34102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 >> 172.21.15.105:0/3034798659 conn(0x7f3f340fc420 msgr2=0x7f3f34104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 shutdown_connections 2024-04-22T06:37:04.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.663+0000 7f3f39723640 1 -- 172.21.15.105:0/3034798659 wait complete. 2024-04-22T06:37:04.672 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:37:04.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 -- 172.21.15.105:0/1740953212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960103190 msgr2=0x7f0960103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 --2- 172.21.15.105:0/1740953212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960103190 0x7f0960103610 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0954009900 tx=0x7f095402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 -- 172.21.15.105:0/1740953212 shutdown_connections 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 --2- 172.21.15.105:0/1740953212 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 0x7f096010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 --2- 172.21.15.105:0/1740953212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960103190 0x7f0960103610 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 --2- 172.21.15.105:0/1740953212 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960101fa0 0x7f09601023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 -- 172.21.15.105:0/1740953212 >> 172.21.15.105:0/1740953212 conn(0x7f09600fd530 msgr2=0x7f09600ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 -- 172.21.15.105:0/1740953212 shutdown_connections 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 -- 172.21.15.105:0/1740953212 wait complete. 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.776+0000 7f096605c640 1 Processor -- start 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- start start 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960103190 0x7f0960074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 0x7f0960074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f096010db40 con 0x7f0960101fa0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d9c0 con 0x7f0960103190 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f096010dcc0 con 0x7f0960103b50 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54088/0 (socket says 172.21.15.105:54088) 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 -- 172.21.15.105:0/3034226944 learned_addr learned my addr 172.21.15.105:0/3034226944 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095ffff640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 0x7f0960074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095effd640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960103190 0x7f0960074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 -- 172.21.15.105:0/3034226944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 msgr2=0x7f0960074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 0x7f0960074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 -- 172.21.15.105:0/3034226944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960103190 msgr2=0x7f0960074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960103190 0x7f0960074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 -- 172.21.15.105:0/3034226944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0960075110 con 0x7f0960101fa0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f095000b450 tx=0x7f095000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095cff9640 1 -- 172.21.15.105:0/3034226944 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950015020 con 0x7f0960101fa0 2024-04-22T06:37:04.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095cff9640 1 -- 172.21.15.105:0/3034226944 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0950011d40 con 0x7f0960101fa0 2024-04-22T06:37:04.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09601a23a0 con 0x7f0960101fa0 2024-04-22T06:37:04.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f095cff9640 1 -- 172.21.15.105:0/3034226944 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095000f340 con 0x7f0960101fa0 2024-04-22T06:37:04.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.777+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09601a29c0 con 0x7f0960101fa0 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.779+0000 7f095cff9640 1 -- 172.21.15.105:0/3034226944 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0950018020 con 0x7f0960101fa0 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 Processor -- start 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095cff9640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 0x7f09480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 -- start start 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095ffff640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 0x7f09480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d424cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095cff9640 1 -- 172.21.15.105:0/3034226944 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f09500cfeb0 con 0x7f0960101fa0 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 0x558a4d42beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d388b10 con 0x558a4d423e40 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4d326240 con 0x558a4d4252d0 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f0968c53740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558a4d384ad0 con 0x558a4d424830 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54090/0 (socket says 172.21.15.105:54090) 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 -- 172.21.15.105:0/3361755627 learned_addr learned my addr 172.21.15.105:0/3361755627 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095ffff640 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 0x558a4d42beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 -- 172.21.15.105:0/3361755627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 msgr2=0x558a4d424cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d424cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 -- 172.21.15.105:0/3361755627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 msgr2=0x558a4d42beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 0x558a4d42beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095f7fe640 1 -- 172.21.15.105:0/3361755627 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4d09b2d0 con 0x558a4d423e40 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.780+0000 7f095ffff640 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 0x558a4d42beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f095ffff640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 0x7f09480a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f095800ad50 tx=0x7f095800e720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f095f7fe640 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0950098c10 tx=0x7f095009ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbc619b1e673a04 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0945ffb640 1 -- 172.21.15.105:0/3361755627 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500988b0 con 0x558a4d423e40 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0945ffb640 1 -- 172.21.15.105:0/3361755627 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0950010500 con 0x558a4d423e40 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0945ffb640 1 -- 172.21.15.105:0/3361755627 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500cddb0 con 0x558a4d423e40 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 -- 172.21.15.105:0/3361755627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 msgr2=0x558a4d424260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0950098c10 tx=0x7f095009ca70 comp rx=0 tx=0).stop 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 -- 172.21.15.105:0/3361755627 shutdown_connections 2024-04-22T06:37:04.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d4252d0 0x558a4d42beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d424cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 --2- 172.21.15.105:0/3361755627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d423e40 0x558a4d424260 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.781+0000 7f0968c53740 1 -- 172.21.15.105:0/3361755627 >> 172.21.15.105:0/3361755627 conn(0x558a4d41ff40 msgr2=0x558a4d41e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- 172.21.15.105:0/3361755627 shutdown_connections 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- 172.21.15.105:0/3361755627 wait complete. 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- start start 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 Processor -- start 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d42c4b0 con 0x558a4d4252d0 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4d311190 con 0x558a4d423e40 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558a4d42c630 con 0x558a4d424830 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f0968c53740 10 client.0 _ll_register_callbacks cb 0x558a4d30c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f095ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f095ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54102/0 (socket says 172.21.15.105:54102) 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f095ffff640 1 -- 172.21.15.105:0/1121489267 learned_addr learned my addr 172.21.15.105:0/1121489267 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f095f7fe640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.782+0000 7f095effd640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 -- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 msgr2=0x558a4d42e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 -- 172.21.15.105:0/1121489267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 msgr2=0x558a4d42dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 -- 172.21.15.105:0/1121489267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4d1656b0 con 0x558a4d4252d0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095effd640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f095ffff640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0958012e00 tx=0x7f0958004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af797834e1a29f81 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958022030 con 0x558a4d4252d0 2024-04-22T06:37:04.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f0958004b20 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0950010500 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f095802e050 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 10 client.15111 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a4d438790 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0950010500 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f095801c040 con 0x558a4d4252d0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 10 client.15111 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 10 client.15111 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 20 client.15111 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d43aec0 0x558a4d43c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 4 client.15111 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.783+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a4d396530 con 0x558a4d43aec0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.784+0000 7f095f7fe640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d43aec0 0x558a4d43c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.784+0000 7f095f7fe640 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d43aec0 0x558a4d43c330 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf5677818e28e309 server_cookie=b186167bee46b721 in_seq=0 out_seq=0 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.784+0000 7f093effd640 10 client.15111 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f093effd640 1 -- 172.21.15.105:0/1121489267 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a4d396530 con 0x558a4d43aec0 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f093effd640 10 client.15111 handle_command_reply: tid=0 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 client.15111 shutdown 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 msgr2=0x558a4d435440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0958012e00 tx=0x7f0958004ab0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 shutdown_connections 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d43aec0 0x558a4d43c330 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435440 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 --2- 172.21.15.105:0/1121489267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 >> 172.21.15.105:0/1121489267 conn(0x558a4d41ff40 msgr2=0x558a4d41da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.785+0000 7f093effd640 0 client.15111 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:04.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.786+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 shutdown_connections 2024-04-22T06:37:04.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.786+0000 7f0968c53740 1 -- 172.21.15.105:0/1121489267 wait complete. 2024-04-22T06:37:04.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.786+0000 7f0968c53740 20 client.15111 trim_cache size 0 max 16384 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 Processor -- start 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 -- start start 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43a670 con 0x558a4d4252d0 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43ad40 con 0x558a4d423e40 2024-04-22T06:37:04.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.803+0000 7f0968c53740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43b410 con 0x558a4d424830 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54114/0 (socket says 172.21.15.105:54114) 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 -- 172.21.15.105:0/2473428795 learned_addr learned my addr 172.21.15.105:0/2473428795 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095f7fe640 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 -- 172.21.15.105:0/2473428795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 msgr2=0x558a4d42c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 -- 172.21.15.105:0/2473428795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 msgr2=0x558a4d42be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 -- 172.21.15.105:0/2473428795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4d41f230 con 0x558a4d4252d0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f095ffff640 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f09580059e0 tx=0x7f0958005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7dbe352cae64ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f09457fa640 1 -- 172.21.15.105:0/2473428795 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095802b070 con 0x558a4d4252d0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f09457fa640 1 -- 172.21.15.105:0/2473428795 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095801e6b0 con 0x558a4d4252d0 2024-04-22T06:37:04.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.804+0000 7f09457fa640 1 -- 172.21.15.105:0/2473428795 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958022040 con 0x558a4d4252d0 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- 172.21.15.105:0/2473428795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 msgr2=0x558a4d435400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f09580059e0 tx=0x7f0958005fa0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- 172.21.15.105:0/2473428795 shutdown_connections 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d4252d0 0x558a4d435400 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d424830 0x558a4d42c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- 172.21.15.105:0/2473428795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d42be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- 172.21.15.105:0/2473428795 >> 172.21.15.105:0/2473428795 conn(0x558a4d41ff40 msgr2=0x558a4d42dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- 172.21.15.105:0/2473428795 shutdown_connections 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- 172.21.15.105:0/2473428795 wait complete. 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- start start 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 Processor -- start 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d4313e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d4252d0 0x558a4d433e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43c1b0 con 0x558a4d424830 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43bae0 con 0x558a4d423e40 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558a4d43c880 con 0x558a4d4252d0 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f0968c53740 10 client.0 _ll_register_callbacks cb 0x7f0958016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f095effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f095effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54122/0 (socket says 172.21.15.105:54122) 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.805+0000 7f095effd640 1 -- 172.21.15.105:0/608189995 learned_addr learned my addr 172.21.15.105:0/608189995 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095f7fe640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d4313e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 -- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d4252d0 msgr2=0x558a4d433e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d4252d0 0x558a4d433e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 -- 172.21.15.105:0/608189995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 msgr2=0x558a4d4313e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d4313e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 -- 172.21.15.105:0/608189995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4d4343b0 con 0x558a4d424830 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f095effd640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0954036040 tx=0x7f0954004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ab6bd8a9c1b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954042070 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f0954005240 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095801e6b0 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f095404e050 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 10 client.15117 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a4d435750 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095801e6b0 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f09540035f0 con 0x558a4d424830 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 10 client.15117 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 10 client.15117 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 20 client.15117 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d4a56a0 0x558a4d4a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 4 client.15117 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:04.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.806+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "status"}) -- 0x558a4d3903a0 con 0x558a4d4a56a0 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f095f7fe640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d4a56a0 0x558a4d4a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f093effd640 10 client.15117 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f095f7fe640 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d4a56a0 0x558a4d4a1ae0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89f6c236a6b86050 server_cookie=ed6a60c710499430 in_seq=0 out_seq=0 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f093effd640 1 -- 172.21.15.105:0/608189995 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558a4d3903a0 con 0x558a4d4a56a0 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f093effd640 10 client.15117 handle_command_reply: tid=0 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.807+0000 7f0968c53740 1 client.15117 shutdown 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 msgr2=0x558a4d433930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0954036040 tx=0x7f0954004270 comp rx=0 tx=0).stop 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 shutdown_connections 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x558a4d4a56a0 0x558a4d4a1ae0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558a4d4252d0 0x558a4d433e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f093effd640 0 client.15117 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558a4d424830 0x558a4d433930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 --2- 172.21.15.105:0/608189995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4d423e40 0x558a4d4313e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 >> 172.21.15.105:0/608189995 conn(0x558a4d41ff40 msgr2=0x558a4d428840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 shutdown_connections 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 1 -- 172.21.15.105:0/608189995 wait complete. 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f0968c53740 20 client.15117 trim_cache size 0 max 16384 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:04.815 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 1, 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24472, 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:04.816 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 81.791116110999994, 2024-04-22T06:37:04.817 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:04.817 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:04.817 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:04.817 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 97.946105759999995 2024-04-22T06:37:04.817 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 msgr2=0x7f09480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 0x7f09480a6180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f095800ad50 tx=0x7f095800e720 comp rx=0 tx=0).stop 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 msgr2=0x7f0960075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f095000b450 tx=0x7f095000b910 comp rx=0 tx=0).stop 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 shutdown_connections 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f09480a3cc0 0x7f09480a6180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0960103b50 0x7f0960074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0960103190 0x7f0960074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 --2- 172.21.15.105:0/3034226944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0960101fa0 0x7f0960075d00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 >> 172.21.15.105:0/3034226944 conn(0x7f09600fd530 msgr2=0x7f09601065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 shutdown_connections 2024-04-22T06:37:04.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.808+0000 7f096605c640 1 -- 172.21.15.105:0/3034226944 wait complete. 2024-04-22T06:37:04.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/580853343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:04.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:04 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1099802977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:04.820 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:37:04.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 -- 172.21.15.105:0/2359052012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0100e60 msgr2=0x7f37f0101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 --2- 172.21.15.105:0/2359052012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0100e60 0x7f37f0101280 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009900 tx=0x7f37e002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 -- 172.21.15.105:0/2359052012 shutdown_connections 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 --2- 172.21.15.105:0/2359052012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0102a00 0x7f37f0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 --2- 172.21.15.105:0/2359052012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102040 0x7f37f01024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 --2- 172.21.15.105:0/2359052012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0100e60 0x7f37f0101280 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 -- 172.21.15.105:0/2359052012 >> 172.21.15.105:0/2359052012 conn(0x7f37f00fc420 msgr2=0x7f37f00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 -- 172.21.15.105:0/2359052012 shutdown_connections 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.920+0000 7f37f4be0640 1 -- 172.21.15.105:0/2359052012 wait complete. 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 Processor -- start 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 -- start start 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 0x7f37f006b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102a00 0x7f37f019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37f010c9f0 con 0x7f37f0102040 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37f010c870 con 0x7f37f0102a00 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37f4be0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f37f010cb70 con 0x7f37f0100e60 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54142/0 (socket says 172.21.15.105:54142) 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 -- 172.21.15.105:0/1902924237 learned_addr learned my addr 172.21.15.105:0/1902924237 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102a00 0x7f37f019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 0x7f37f006b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 -- 172.21.15.105:0/1902924237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 msgr2=0x7f37f006b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 0x7f37f006b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 -- 172.21.15.105:0/1902924237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102a00 msgr2=0x7f37f019f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102a00 0x7f37f019f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37edd9b640 1 -- 172.21.15.105:0/1902924237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f019fe40 con 0x7f37f0102040 2024-04-22T06:37:04.940 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.921+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 0x7f37f006b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f37e400d950 tx=0x7f37e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37df7fe640 1 -- 172.21.15.105:0/1902924237 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4017070 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37df7fe640 1 -- 172.21.15.105:0/1902924237 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e4009c30 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37df7fe640 1 -- 172.21.15.105:0/1902924237 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e400f620 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f01a0130 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.922+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f01a06d0 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.924+0000 7f37df7fe640 1 -- 172.21.15.105:0/1902924237 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f37e4010460 con 0x7f37f0102040 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.924+0000 7f37f77cd740 1 Processor -- start 2024-04-22T06:37:04.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 -- start start 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a283c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a283d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654a279cb10 con 0x5654a283d6e0 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5654a273a240 con 0x5654a283c250 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37f77cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654a2798ad0 con 0x5654a283cc40 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54148/0 (socket says 172.21.15.105:54148) 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 -- 172.21.15.105:0/1950654851 learned_addr learned my addr 172.21.15.105:0/1950654851 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37df7fe640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 0x7f37d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a283c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a283d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 0x7f37d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 -- 172.21.15.105:0/1950654851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 msgr2=0x5654a283d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a283d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 -- 172.21.15.105:0/1950654851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 msgr2=0x5654a283c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a283c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37df7fe640 1 -- 172.21.15.105:0/1902924237 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f37e40d0140 con 0x7f37f0102040 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 -- 172.21.15.105:0/1950654851 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a24af2d0 con 0x5654a283d6e0 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a283c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.925+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37e40cdb20 tx=0x7f37e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6f4817e01a83a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37dcff9640 1 -- 172.21.15.105:0/1950654851 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4020070 con 0x5654a283d6e0 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 0x7f37d80a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5654a2833200 tx=0x7f37e8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37dcff9640 1 -- 172.21.15.105:0/1950654851 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e4009de0 con 0x5654a283d6e0 2024-04-22T06:37:04.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37dcff9640 1 -- 172.21.15.105:0/1950654851 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40140d0 con 0x5654a283d6e0 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 -- 172.21.15.105:0/1950654851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 msgr2=0x5654a28442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37e40cdb20 tx=0x7f37e4005880 comp rx=0 tx=0).stop 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 -- 172.21.15.105:0/1950654851 shutdown_connections 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a28442c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a283d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1950654851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a283c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 -- 172.21.15.105:0/1950654851 >> 172.21.15.105:0/1950654851 conn(0x5654a2833f40 msgr2=0x5654a2832490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 -- 172.21.15.105:0/1950654851 shutdown_connections 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.926+0000 7f37f77cd740 1 -- 172.21.15.105:0/1950654851 wait complete. 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 -- start start 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 Processor -- start 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283cc40 0x5654a2846770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a284d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654a2844860 con 0x5654a283c250 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5654a2725190 con 0x5654a283cc40 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654a28449e0 con 0x5654a283d6e0 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37f77cd740 10 client.0 _ll_register_callbacks cb 0x5654a27208b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54152/0 (socket says 172.21.15.105:54152) 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 -- 172.21.15.105:0/3732608738 learned_addr learned my addr 172.21.15.105:0/3732608738 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283cc40 0x5654a2846770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37eed9d640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a284d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 -- 172.21.15.105:0/3732608738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 msgr2=0x5654a284d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a284d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 -- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283cc40 msgr2=0x5654a2846770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283cc40 0x5654a2846770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 -- 172.21.15.105:0/3732608738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a25796b0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37eed9d640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a284d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37ee59c640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009900 tx=0x7f37e00042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3570361c5e20f3da server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.927+0000 7f37d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e0005450 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f37e00046c0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e4009de0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f37e00055f0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 10 client.15129 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5654a2850be0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e4009de0 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f37e0003940 con 0x5654a283c250 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 10 client.15129 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 20 client.15129 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a2854710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 4 client.15129 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654a27aa530 con 0x5654a28532a0 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.928+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a2854710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.929+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a2854710 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f17d5f9cf648277 server_cookie=3ba073bc579baa73 in_seq=0 out_seq=0 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.929+0000 7f37d5ffb640 10 client.15129 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.929+0000 7f37d5ffb640 1 -- 172.21.15.105:0/3732608738 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654a27aa530 con 0x5654a28532a0 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37d5ffb640 10 client.15129 handle_command_reply: tid=0 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 client.15129 shutdown 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 msgr2=0x5654a2846230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009900 tx=0x7f37e00042d0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 shutdown_connections 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a2854710 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a284d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37d5ffb640 0 client.15129 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283cc40 0x5654a2846770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3732608738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283c250 0x5654a2846230 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 >> 172.21.15.105:0/3732608738 conn(0x5654a2833f40 msgr2=0x5654a2831a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 shutdown_connections 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 1 -- 172.21.15.105:0/3732608738 wait complete. 2024-04-22T06:37:04.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.930+0000 7f37f77cd740 20 client.15129 trim_cache size 0 max 16384 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 Processor -- start 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 -- start start 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a28310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a2831620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654a2852a50 con 0x5654a283cc40 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5654a2853120 con 0x5654a283c250 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37f77cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654a28537f0 con 0x5654a283d6e0 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54162/0 (socket says 172.21.15.105:54162) 2024-04-22T06:37:04.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 -- 172.21.15.105:0/3850068432 learned_addr learned my addr 172.21.15.105:0/3850068432 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 -- 172.21.15.105:0/3850068432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 msgr2=0x5654a2831620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a2831620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 -- 172.21.15.105:0/3850068432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 msgr2=0x5654a28310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a28310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 -- 172.21.15.105:0/3850068432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a2833230 con 0x5654a283cc40 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37edd9b640 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f37e4098c90 tx=0x7f37e4097f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59757e1b7b7fe8cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37d7fff640 1 -- 172.21.15.105:0/3850068432 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4020040 con 0x5654a283cc40 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37d7fff640 1 -- 172.21.15.105:0/3850068432 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e409c9c0 con 0x5654a283cc40 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.948+0000 7f37d7fff640 1 -- 172.21.15.105:0/3850068432 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4098270 con 0x5654a283cc40 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- 172.21.15.105:0/3850068432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 msgr2=0x5654a284d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f37e4098c90 tx=0x7f37e4097f30 comp rx=0 tx=0).stop 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- 172.21.15.105:0/3850068432 shutdown_connections 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283d6e0 0x5654a2831620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283cc40 0x5654a284d770 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- 172.21.15.105:0/3850068432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a28310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- 172.21.15.105:0/3850068432 >> 172.21.15.105:0/3850068432 conn(0x5654a2833f40 msgr2=0x5654a2840bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- 172.21.15.105:0/3850068432 shutdown_connections 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- 172.21.15.105:0/3850068432 wait complete. 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- start start 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 Processor -- start 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a284b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a284be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654a2854590 con 0x5654a283d6e0 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5654a2853ec0 con 0x5654a283c250 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5654a2854c60 con 0x5654a283cc40 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37f77cd740 10 client.0 _ll_register_callbacks cb 0x7f37e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37eed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.949+0000 7f37eed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54176/0 (socket says 172.21.15.105:54176) 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 -- 172.21.15.105:0/1599980803 learned_addr learned my addr 172.21.15.105:0/1599980803 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37edd9b640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a284be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 -- 172.21.15.105:0/1599980803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 msgr2=0x5654a284be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a284be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a284b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 -- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 msgr2=0x5654a284b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a284b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 -- 172.21.15.105:0/1599980803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a284ccb0 con 0x5654a283d6e0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37eed9d640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f37e80108e0 tx=0x7f37e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2991c5779c5bc2e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8018070 con 0x5654a283d6e0 2024-04-22T06:37:04.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f37e801d820 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e409c9c0 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f37e801da20 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37f77cd740 10 client.15135 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5654a28b95d0 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e409c9c0 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.950+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f37e8025070 con 0x5654a283d6e0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 10 client.15135 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 10 client.15135 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 20 client.15135 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a28bcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 4 client.15135 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "status"}) -- 0x5654a27a43a0 con 0x5654a28532a0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a28bcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37ee59c640 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a28bcf10 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f1082bc2dc4186 server_cookie=9249d455910182d0 in_seq=0 out_seq=0 2024-04-22T06:37:04.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37d5ffb640 10 client.15135 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37d5ffb640 1 -- 172.21.15.105:0/1599980803 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5654a27a43a0 con 0x5654a28532a0 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37d5ffb640 10 client.15135 handle_command_reply: tid=0 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 client.15135 shutdown 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 msgr2=0x5654a284c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f37e80108e0 tx=0x7f37e8009b50 comp rx=0 tx=0).stop 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.951+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 shutdown_connections 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5654a28532a0 0x5654a28bcf10 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654a283d6e0 0x5654a284c480 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5654a283cc40 0x5654a284be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37d5ffb640 0 client.15135 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 --2- 172.21.15.105:0/1599980803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654a283c250 0x5654a284b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 >> 172.21.15.105:0/1599980803 conn(0x5654a2833f40 msgr2=0x5654a2836c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 shutdown_connections 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 1 -- 172.21.15.105:0/1599980803 wait complete. 2024-04-22T06:37:04.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f77cd740 20 client.15135 trim_cache size 0 max 16384 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 0, 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24500, 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 84.147485289000002, 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:04.961 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:04.962 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 89.667721080000007 2024-04-22T06:37:04.962 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:04.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 msgr2=0x7f37d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 0x7f37d80a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5654a2833200 tx=0x7f37e8004ab0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 msgr2=0x7f37f019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f37e400d950 tx=0x7f37e400de10 comp rx=0 tx=0).stop 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 shutdown_connections 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f37d80a3cc0 0x7f37d80a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f0102a00 0x7f37f019f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37f0102040 0x7f37f019cf40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 --2- 172.21.15.105:0/1902924237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f37f0100e60 0x7f37f006b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 >> 172.21.15.105:0/1902924237 conn(0x7f37f00fc420 msgr2=0x7f37f01054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:04.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 shutdown_connections 2024-04-22T06:37:04.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:04.952+0000 7f37f4be0640 1 -- 172.21.15.105:0/1902924237 wait complete. 2024-04-22T06:37:04.965 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T06:37:04.966 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:37:04.976 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:04.977 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-22T06:37:04.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/580853343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:04.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:04 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1099802977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:05.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/580853343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:05.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:04 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1099802977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:05.210 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:05.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:05 smithi105 ceph-mon[27916]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:05.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:05 smithi177 ceph-mon[31346]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:06.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:05 smithi112 ceph-mon[31236]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:06.806 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 -- 172.21.15.105:0/1085671678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440104660 msgr2=0x7f344010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:06.806 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 --2- 172.21.15.105:0/1085671678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440104660 0x7f344010af10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f343800b0d0 tx=0x7f343800b3e0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 -- 172.21.15.105:0/1085671678 shutdown_connections 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 --2- 172.21.15.105:0/1085671678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440104660 0x7f344010af10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 --2- 172.21.15.105:0/1085671678 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440102ac0 0x7f3440102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 --2- 172.21.15.105:0/1085671678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440103cc0 0x7f3440104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 -- 172.21.15.105:0/1085671678 >> 172.21.15.105:0/1085671678 conn(0x7f34400fe060 msgr2=0x7f34401004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 -- 172.21.15.105:0/1085671678 shutdown_connections 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 -- 172.21.15.105:0/1085671678 wait complete. 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.806+0000 7f3445117700 1 Processor -- start 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 -- start start 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 0x7f3440199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440104660 0x7f34401a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f344010e540 con 0x7f3440102ac0 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f344010e3c0 con 0x7f3440104660 2024-04-22T06:37:06.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f3445117700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f344010e6c0 con 0x7f3440103cc0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54184/0 (socket says 172.21.15.105:54184) 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 -- 172.21.15.105:0/590956918 learned_addr learned my addr 172.21.15.105:0/590956918 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343f59e700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440104660 0x7f34401a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343e59c700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 0x7f3440199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 -- 172.21.15.105:0/590956918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 msgr2=0x7f3440199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 0x7f3440199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 -- 172.21.15.105:0/590956918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440104660 msgr2=0x7f34401a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440104660 0x7f34401a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 -- 172.21.15.105:0/590956918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34401a0720 con 0x7f3440102ac0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343e59c700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 0x7f3440199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.807+0000 7f343ed9d700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f343000d9b0 tx=0x7f343000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.808+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430017030 con 0x7f3440102ac0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.808+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34401a0a10 con 0x7f3440102ac0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.808+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34300044d0 con 0x7f3440102ac0 2024-04-22T06:37:06.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.808+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34300039e0 con 0x7f3440102ac0 2024-04-22T06:37:06.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.808+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34401a0fa0 con 0x7f3440102ac0 2024-04-22T06:37:06.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.809+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3440069730 con 0x7f3440102ac0 2024-04-22T06:37:06.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.810+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f343001a070 con 0x7f3440102ac0 2024-04-22T06:37:06.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.810+0000 7f3427fff700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 0x7f34280a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:06.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.811+0000 7f343e59c700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 0x7f34280a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:06.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.811+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f34300d0170 con 0x7f3440102ac0 2024-04-22T06:37:06.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.811+0000 7f343e59c700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 0x7f34280a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3434005fd0 tx=0x7f3434005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:06.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:06.814+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f343000e350 con 0x7f3440102ac0 2024-04-22T06:37:07.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.031+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f3440062e80 con 0x7f34280a3cd0 2024-04-22T06:37:07.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.032+0000 7f3427fff700 1 -- 172.21.15.105:0/590956918 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f3440062e80 con 0x7f34280a3cd0 2024-04-22T06:37:07.030 INFO:teuthology.orchestra.run.smithi105.stdout:Backend: cephadm 2024-04-22T06:37:07.030 INFO:teuthology.orchestra.run.smithi105.stdout:Available: Yes 2024-04-22T06:37:07.030 INFO:teuthology.orchestra.run.smithi105.stdout:Paused: No 2024-04-22T06:37:07.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 msgr2=0x7f34280a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:07.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 0x7f34280a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3434005fd0 tx=0x7f3434005e20 comp rx=0 tx=0).stop 2024-04-22T06:37:07.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 msgr2=0x7f3440198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:07.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f343000d9b0 tx=0x7f343000dcc0 comp rx=0 tx=0).stop 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 shutdown_connections 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f34280a3cd0 0x7f34280a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3440104660 0x7f34401a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3440102ac0 0x7f3440198ac0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 --2- 172.21.15.105:0/590956918 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3440103cc0 0x7f3440199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 >> 172.21.15.105:0/590956918 conn(0x7f34400fe060 msgr2=0x7f3440106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 shutdown_connections 2024-04-22T06:37:07.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:07.034+0000 7f3445117700 1 -- 172.21.15.105:0/590956918 wait complete. 2024-04-22T06:37:07.744 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-22T06:37:07.985 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:07.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:07 smithi177 ceph-mon[31346]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:08.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:07 smithi105 ceph-mon[27916]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:08.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:07 smithi112 ceph-mon[31236]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:08.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:08 smithi177 ceph-mon[31346]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:09.061 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:08 smithi105 ceph-mon[27916]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:09.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:08 smithi112 ceph-mon[31236]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- 172.21.15.105:0/3594835999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 msgr2=0x7f14c4104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 --2- 172.21.15.105:0/3594835999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4104100 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f14b40099c0 tx=0x7f14b4009cd0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- 172.21.15.105:0/3594835999 shutdown_connections 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 --2- 172.21.15.105:0/3594835999 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4104640 0x7f14c410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 --2- 172.21.15.105:0/3594835999 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4102aa0 0x7f14c4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 --2- 172.21.15.105:0/3594835999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4104100 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- 172.21.15.105:0/3594835999 >> 172.21.15.105:0/3594835999 conn(0x7f14c40fe040 msgr2=0x7f14c4100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- 172.21.15.105:0/3594835999 shutdown_connections 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- 172.21.15.105:0/3594835999 wait complete. 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 Processor -- start 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.627+0000 7f14cac6b700 1 -- start start 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4102aa0 0x7f14c4198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4104640 0x7f14c41a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f14c410e510 con 0x7f14c4103ca0 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14c410e390 con 0x7f14c4102aa0 2024-04-22T06:37:09.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f14c410e690 con 0x7f14c4104640 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54200/0 (socket says 172.21.15.105:54200) 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 -- 172.21.15.105:0/2138642844 learned_addr learned my addr 172.21.15.105:0/2138642844 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14c9208700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4104640 0x7f14c41a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14c8a07700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4102aa0 0x7f14c4198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 -- 172.21.15.105:0/2138642844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4104640 msgr2=0x7f14c41a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4104640 0x7f14c41a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 -- 172.21.15.105:0/2138642844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4102aa0 msgr2=0x7f14c4198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4102aa0 0x7f14c4198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 -- 172.21.15.105:0/2138642844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c41a0820 con 0x7f14c4103ca0 2024-04-22T06:37:09.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14bbfff700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005910 tx=0x7f14b4005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4027070 con 0x7f14c4103ca0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c41a0ab0 con 0x7f14c4103ca0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14b4004020 con 0x7f14c4103ca0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.628+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4022c50 con 0x7f14c4103ca0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.629+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c41a0fe0 con 0x7f14c4103ca0 2024-04-22T06:37:09.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.630+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c4069730 con 0x7f14c4103ca0 2024-04-22T06:37:09.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.631+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f14b402a070 con 0x7f14c4103ca0 2024-04-22T06:37:09.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.631+0000 7f14b9ffb700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 0x7f14ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:09.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.631+0000 7f14c8a07700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 0x7f14ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:09.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.631+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f14b40d1aa0 con 0x7f14c4103ca0 2024-04-22T06:37:09.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.632+0000 7f14c8a07700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 0x7f14ac0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14c4103b00 tx=0x7f14c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:09.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.635+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14b4031070 con 0x7f14c4103ca0 2024-04-22T06:37:09.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.848+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f14c4062e80 con 0x7f14ac0a3cd0 2024-04-22T06:37:09.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.861+0000 7f14b9ffb700 1 -- 172.21.15.105:0/2138642844 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f14c4062e80 con 0x7f14ac0a3cd0 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.a smithi105 running (97s) 15s ago 97s 19.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 4dff5045448c 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.b smithi112 running (94s) 28s ago 94s 25.7M - 19.0.0-3156-g86f7587a 1b5e03f260bd 9d0f7a3d9909 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.c smithi177 running (108s) 28s ago 108s 19.1M - 19.0.0-3156-g86f7587a 1b5e03f260bd 26eafc33cbaa 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.d smithi105 running (116s) 15s ago 116s 13.2M - 19.0.0-3156-g86f7587a 1b5e03f260bd 5a6602ba459c 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.e smithi112 running (113s) 28s ago 113s 15.9M - 19.0.0-3156-g86f7587a 1b5e03f260bd bd5a881cbc55 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.f smithi177 running (111s) 28s ago 110s 20.3M - 19.0.0-3156-g86f7587a 1b5e03f260bd b63431b0a6a1 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.g smithi105 running (100s) 15s ago 100s 16.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 21f47b3cd334 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.h smithi112 running (105s) 28s ago 105s 17.8M - 19.0.0-3156-g86f7587a 1b5e03f260bd ae424537b9fb 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.i smithi177 running (103s) 28s ago 102s 20.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 33163aa97e77 2024-04-22T06:37:09.860 INFO:teuthology.orchestra.run.smithi105.stdout:mds.j smithi105 running (119s) 15s ago 118s 16.2M - 19.0.0-3156-g86f7587a 1b5e03f260bd b85b65857577 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mds.k smithi112 running (2m) 28s ago 2m 16.9M - 19.0.0-3156-g86f7587a 1b5e03f260bd 34e012d0142e 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mds.l smithi177 running (2m) 28s ago 2m 16.0M - 19.0.0-3156-g86f7587a 1b5e03f260bd 75448994ff9a 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.x smithi105 *:9283,8765 running (9m) 15s ago 9m 514M - 19.0.0-3156-g86f7587a 1b5e03f260bd f0a31be559b2 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.y smithi112 *:8443,8765 running (7m) 28s ago 7m 451M - 19.0.0-3156-g86f7587a 1b5e03f260bd ed5a0a9e12a6 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mgr.z smithi177 *:8443,8765 running (7m) 28s ago 7m 451M - 19.0.0-3156-g86f7587a 1b5e03f260bd a04243b094b2 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mon.a smithi105 running (9m) 15s ago 9m 52.8M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 0d1415368ca0 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mon.b smithi112 running (7m) 28s ago 7m 40.1M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd 8c86d0b36e38 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:mon.c smithi177 running (7m) 28s ago 7m 39.8M 2048M 19.0.0-3156-g86f7587a 1b5e03f260bd f0e1fc003780 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 smithi105 running (6m) 15s ago 6m 56.8M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd c8a378707c29 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 smithi112 running (6m) 28s ago 6m 49.2M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 7a0008614c63 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 smithi177 running (6m) 28s ago 6m 50.2M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd bec8fdf72983 2024-04-22T06:37:09.861 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 smithi105 running (5m) 15s ago 5m 47.5M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd a1352d1bbb3a 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 smithi112 running (5m) 28s ago 5m 44.6M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd c12e645b8606 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 smithi177 running (4m) 28s ago 4m 58.7M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 317037bfe4f8 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6 smithi105 running (4m) 15s ago 4m 53.6M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd a55b21bfd020 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7 smithi112 running (4m) 28s ago 4m 46.1M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 0560ac1df4e4 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.8 smithi177 running (3m) 28s ago 3m 55.9M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 0107623d05b8 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.9 smithi105 running (3m) 15s ago 3m 52.9M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 4cb18d0dc259 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.10 smithi112 running (2m) 28s ago 2m 56.0M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd 0951a4e21c93 2024-04-22T06:37:09.862 INFO:teuthology.orchestra.run.smithi105.stdout:osd.11 smithi177 running (2m) 28s ago 2m 43.2M 4272M 19.0.0-3156-g86f7587a 1b5e03f260bd b3c50409993e 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 msgr2=0x7f14ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 0x7f14ac0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14c4103b00 tx=0x7f14c000b410 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 msgr2=0x7f14c4199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f14b4005910 tx=0x7f14b4005b00 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 shutdown_connections 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f14ac0a3cd0 0x7f14ac0a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14c4102aa0 0x7f14c4198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f14c4104640 0x7f14c41a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 --2- 172.21.15.105:0/2138642844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14c4103ca0 0x7f14c4199100 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 >> 172.21.15.105:0/2138642844 conn(0x7f14c40fe040 msgr2=0x7f14c4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 shutdown_connections 2024-04-22T06:37:09.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:09.864+0000 7f14cac6b700 1 -- 172.21.15.105:0/2138642844 wait complete. 2024-04-22T06:37:09.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:09 smithi177 ceph-mon[31346]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:37:10.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:09 smithi105 ceph-mon[27916]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:37:10.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:09 smithi112 ceph-mon[31236]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-22T06:37:10.572 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-22T06:37:10.825 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:11 smithi177 ceph-mon[31346]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:11.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:11 smithi177 ceph-mon[31346]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:12.120 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:11 smithi105 ceph-mon[27916]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:12.121 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:11 smithi105 ceph-mon[27916]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:12.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:11 smithi112 ceph-mon[31236]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:12.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:11 smithi112 ceph-mon[31236]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 -- 172.21.15.105:0/3939327284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 msgr2=0x7f5090101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 --2- 172.21.15.105:0/3939327284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090101db0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f50880099c0 tx=0x7f5088009cd0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 -- 172.21.15.105:0/3939327284 shutdown_connections 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 --2- 172.21.15.105:0/3939327284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090103530 0x7f5090109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 --2- 172.21.15.105:0/3939327284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090101db0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 --2- 172.21.15.105:0/3939327284 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090102b90 0x7f5090102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 -- 172.21.15.105:0/3939327284 >> 172.21.15.105:0/3939327284 conn(0x7f50900fcf30 msgr2=0x7f50900ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 -- 172.21.15.105:0/3939327284 shutdown_connections 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.501+0000 7f5094f67700 1 -- 172.21.15.105:0/3939327284 wait complete. 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 Processor -- start 2024-04-22T06:37:12.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 -- start start 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090102b90 0x7f5090199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090103530 0x7f50901a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f509010d4a0 con 0x7f5090101990 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f509010d320 con 0x7f5090102b90 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f5094f67700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f509010d620 con 0x7f5090103530 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54210/0 (socket says 172.21.15.105:54210) 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 -- 172.21.15.105:0/3940177155 learned_addr learned my addr 172.21.15.105:0/3940177155 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508ed9d700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090103530 0x7f50901a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508dd9b700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090102b90 0x7f5090199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 -- 172.21.15.105:0/3940177155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090103530 msgr2=0x7f50901a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090103530 0x7f50901a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 -- 172.21.15.105:0/3940177155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090102b90 msgr2=0x7f5090199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090102b90 0x7f5090199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 -- 172.21.15.105:0/3940177155 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50901a0890 con 0x7f5090101990 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.502+0000 7f508e59c700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.503+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5088027070 con 0x7f5090101990 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.503+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5088004120 con 0x7f5090101990 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.503+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508800f460 con 0x7f5090101990 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.503+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50901a0b20 con 0x7f5090101990 2024-04-22T06:37:12.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.503+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50901a0ff0 con 0x7f5090101990 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.504+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5090069760 con 0x7f5090101990 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.505+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f508802a070 con 0x7f5090101990 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.505+0000 7f50877fe700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 0x7f507c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.505+0000 7f508dd9b700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 0x7f507c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.505+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f50880d14d0 con 0x7f5090101990 2024-04-22T06:37:12.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.506+0000 7f508dd9b700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 0x7f507c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5078005950 tx=0x7f50780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:12.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.510+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f508805eff0 con 0x7f5090101990 2024-04-22T06:37:12.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.720+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f509006b6c0 con 0x7f507c0a3cd0 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.726+0000 7f50877fe700 1 -- 172.21.15.105:0/3940177155 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f509006b6c0 con 0x7f507c0a3cd0 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stdout:mds.all 12/12 31s ago 2m smithi105=a;smithi105=d;smithi105=g;smithi105=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi177=c;smithi177=f;smithi177=i;smithi177=l;count:12 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stdout:mgr 3/3 31s ago 7m smithi105=x;smithi112=y;smithi177=z;count:3 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stdout:mon 3/3 31s ago 7m smithi105:172.21.15.105=a;smithi112:172.21.15.112=b;smithi177:172.21.15.177=c;count:3 2024-04-22T06:37:12.725 INFO:teuthology.orchestra.run.smithi105.stdout:osd 12 31s ago - 2024-04-22T06:37:12.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 msgr2=0x7f507c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:12.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 0x7f507c0a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5078005950 tx=0x7f50780058e0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 msgr2=0x7f5090198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:12.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088009f80 comp rx=0 tx=0).stop 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 shutdown_connections 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f507c0a3cd0 0x7f507c0a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5090103530 0x7f50901a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5090101990 0x7f5090198c40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.728+0000 7f5094f67700 1 --2- 172.21.15.105:0/3940177155 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5090102b90 0x7f5090199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.729+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 >> 172.21.15.105:0/3940177155 conn(0x7f50900fcf30 msgr2=0x7f509006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.729+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 shutdown_connections 2024-04-22T06:37:12.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:12.729+0000 7f5094f67700 1 -- 172.21.15.105:0/3940177155 wait complete. 2024-04-22T06:37:13.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-22T06:37:13.674 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:14.045 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:13 smithi105 ceph-mon[27916]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:14.045 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:13 smithi105 ceph-mon[27916]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:14.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:13 smithi112 ceph-mon[31236]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:14.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:13 smithi112 ceph-mon[31236]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:14.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:13 smithi177 ceph-mon[31346]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-22T06:37:14.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:13 smithi177 ceph-mon[31346]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- 172.21.15.105:0/2579547049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100102b90 msgr2=0x7fc100102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- 172.21.15.105:0/2579547049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100102b90 0x7fc100102ff0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0099c0 tx=0x7fc0ec009cd0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- 172.21.15.105:0/2579547049 shutdown_connections 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- 172.21.15.105:0/2579547049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc100101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- 172.21.15.105:0/2579547049 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100103530 0x7fc100109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- 172.21.15.105:0/2579547049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100102b90 0x7fc100102ff0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- 172.21.15.105:0/2579547049 >> 172.21.15.105:0/2579547049 conn(0x7fc1000fcf30 msgr2=0x7fc1000ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- 172.21.15.105:0/2579547049 shutdown_connections 2024-04-22T06:37:15.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- 172.21.15.105:0/2579547049 wait complete. 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 Processor -- start 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- start start 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc1001058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 0x7fc100103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d480 con 0x7fc100103530 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d300 con 0x7fc100101990 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc106f7c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc10010d600 con 0x7fc100102b90 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.157+0000 7fc105519700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51588/0 (socket says 172.21.15.105:51588) 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 -- 172.21.15.105:0/3265976764 learned_addr learned my addr 172.21.15.105:0/3265976764 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc104d18700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc1001058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:15.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc0fffff700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 0x7fc100103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 -- 172.21.15.105:0/3265976764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 msgr2=0x7fc100103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 0x7fc100103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 -- 172.21.15.105:0/3265976764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 msgr2=0x7fc1001058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc1001058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 -- 172.21.15.105:0/3265976764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc100104d80 con 0x7fc100103530 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc104d18700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc1001058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc0fffff700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 0x7fc100103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc105519700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800b970 tx=0x7fc0f800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8019070 con 0x7fc100103530 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0f800cde0 con 0x7fc100103530 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8007370 con 0x7fc100103530 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.158+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1001a2df0 con 0x7fc100103530 2024-04-22T06:37:15.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.159+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1001a32c0 con 0x7fc100103530 2024-04-22T06:37:15.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.159+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc100069730 con 0x7fc100103530 2024-04-22T06:37:15.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.160+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fc0f801c070 con 0x7fc100103530 2024-04-22T06:37:15.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.161+0000 7fc0fdffb700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 0x7fc0f00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:15.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.161+0000 7fc104d18700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 0x7fc0f00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:15.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.161+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc0f80cf770 con 0x7fc100103530 2024-04-22T06:37:15.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.161+0000 7fc104d18700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 0x7fc0f00a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1001029f0 tx=0x7fc0f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:15.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.166+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0f809c1f0 con 0x7fc100103530 2024-04-22T06:37:15.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.367+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc1000ff150 con 0x7fc0f00a3cc0 2024-04-22T06:37:15.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.368+0000 7fc0fdffb700 1 -- 172.21.15.105:0/3265976764 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fc1000ff150 con 0x7fc0f00a3cc0 2024-04-22T06:37:15.367 INFO:teuthology.orchestra.run.smithi105.stdout:HOST ADDR LABELS STATUS 2024-04-22T06:37:15.367 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105 172.21.15.105 2024-04-22T06:37:15.367 INFO:teuthology.orchestra.run.smithi105.stdout:smithi112 172.21.15.112 2024-04-22T06:37:15.367 INFO:teuthology.orchestra.run.smithi105.stdout:smithi177 172.21.15.177 2024-04-22T06:37:15.368 INFO:teuthology.orchestra.run.smithi105.stdout:3 hosts in cluster 2024-04-22T06:37:15.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.370+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 msgr2=0x7fc0f00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:15.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.370+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 0x7fc0f00a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1001029f0 tx=0x7fc0f400b410 comp rx=0 tx=0).stop 2024-04-22T06:37:15.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 msgr2=0x7fc1001044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800b970 tx=0x7fc0f800bc80 comp rx=0 tx=0).stop 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 shutdown_connections 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc0f00a3cc0 0x7fc0f00a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc100101990 0x7fc1001058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc100103530 0x7fc1001044b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 --2- 172.21.15.105:0/3265976764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc100102b90 0x7fc100103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 >> 172.21.15.105:0/3265976764 conn(0x7fc1000fcf30 msgr2=0x7fc1000fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 shutdown_connections 2024-04-22T06:37:15.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:15.371+0000 7fc106f7c700 1 -- 172.21.15.105:0/3265976764 wait complete. 2024-04-22T06:37:16.006 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:15 smithi105 ceph-mon[27916]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:16.006 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:15 smithi105 ceph-mon[27916]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:37:16.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-22T06:37:16.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:15 smithi112 ceph-mon[31236]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:16.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:15 smithi112 ceph-mon[31236]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:37:16.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:15 smithi177 ceph-mon[31346]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-22T06:37:16.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:15 smithi177 ceph-mon[31346]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-22T06:37:16.273 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:17.102 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:16 smithi105 ceph-mon[27916]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:17.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:16 smithi112 ceph-mon[31236]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:17.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:16 smithi177 ceph-mon[31346]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:17.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 -- 172.21.15.105:0/5157487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c41019d0 msgr2=0x7f75c4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:17.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 --2- 172.21.15.105:0/5157487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c41019d0 0x7f75c4101df0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f75c000b0d0 tx=0x7f75c000b3e0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 -- 172.21.15.105:0/5157487 shutdown_connections 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 --2- 172.21.15.105:0/5157487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c4103570 0x7f75c4109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 --2- 172.21.15.105:0/5157487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c41019d0 0x7f75c4101df0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 --2- 172.21.15.105:0/5157487 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 0x7f75c4103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 -- 172.21.15.105:0/5157487 >> 172.21.15.105:0/5157487 conn(0x7f75c40fcf90 msgr2=0x7f75c40ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 -- 172.21.15.105:0/5157487 shutdown_connections 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.888+0000 7f75cb55e700 1 -- 172.21.15.105:0/5157487 wait complete. 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 Processor -- start 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 -- start start 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 0x7f75c4105910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 0x7f75c4103fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f75c410d440 con 0x7f75c4103570 2024-04-22T06:37:17.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f75c410d2c0 con 0x7f75c41019d0 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75cb55e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f75c410d5c0 con 0x7f75c4102bd0 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51602/0 (socket says 172.21.15.105:51602) 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 -- 172.21.15.105:0/3872949198 learned_addr learned my addr 172.21.15.105:0/3872949198 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c8af9700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 0x7f75c4103fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c92fa700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 0x7f75c4105910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 -- 172.21.15.105:0/3872949198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 msgr2=0x7f75c4103fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 0x7f75c4103fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 -- 172.21.15.105:0/3872949198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 msgr2=0x7f75c4105910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 0x7f75c4105910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c9afb700 1 -- 172.21.15.105:0/3872949198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c4104dc0 con 0x7f75c4103570 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.889+0000 7f75c92fa700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 0x7f75c4105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:17.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75c9afb700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00e940 tx=0x7f75bc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:17.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc00cbd0 con 0x7f75c4103570 2024-04-22T06:37:17.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75bc00ee90 con 0x7f75c4103570 2024-04-22T06:37:17.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c41a2d80 con 0x7f75c4103570 2024-04-22T06:37:17.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc005590 con 0x7f75c4103570 2024-04-22T06:37:17.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.890+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c41a31d0 con 0x7f75c4103570 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.891+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c4069760 con 0x7f75c4103570 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.892+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f75bc004750 con 0x7f75c4103570 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.892+0000 7f75ba7fc700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 0x7f75b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.893+0000 7f75c92fa700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 0x7f75b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.893+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f75bc0d07a0 con 0x7f75c4103570 2024-04-22T06:37:17.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.893+0000 7f75c92fa700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 0x7f75b00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009be0 tx=0x7f75c000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:17.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:17.896+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75bc053ff0 con 0x7f75c4103570 2024-04-22T06:37:18.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.102+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f75c40ff180 con 0x7f75b00a3c40 2024-04-22T06:37:18.101 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:17 smithi105 ceph-mon[27916]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T06:37:18.103 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.104+0000 7f75ba7fc700 1 -- 172.21.15.105:0/3872949198 <== mgr.14152 v2:172.21.15.105:6800/2556874834 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f75c40ff180 con 0x7f75b00a3c40 2024-04-22T06:37:18.103 INFO:teuthology.orchestra.run.smithi105.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-22T06:37:18.103 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-22T06:37:18.103 INFO:teuthology.orchestra.run.smithi105.stdout:smithi112 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T06:37:18.103 INFO:teuthology.orchestra.run.smithi105.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-22T06:37:18.106 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 msgr2=0x7f75b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:18.106 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 0x7f75b00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009be0 tx=0x7f75c000b040 comp rx=0 tx=0).stop 2024-04-22T06:37:18.106 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 msgr2=0x7f75c41044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00e940 tx=0x7f75bc00ec50 comp rx=0 tx=0).stop 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 shutdown_connections 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f75b00a3c40 0x7f75b00a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75c41019d0 0x7f75c4105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75c4103570 0x7f75c41044f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 --2- 172.21.15.105:0/3872949198 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f75c4102bd0 0x7f75c4103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.107+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 >> 172.21.15.105:0/3872949198 conn(0x7f75c40fcf90 msgr2=0x7f75c40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.108+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 shutdown_connections 2024-04-22T06:37:18.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:18.108+0000 7f75cb55e700 1 -- 172.21.15.105:0/3872949198 wait complete. 2024-04-22T06:37:18.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:17 smithi112 ceph-mon[31236]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T06:37:18.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:17 smithi177 ceph-mon[31346]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-22T06:37:18.800 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T06:37:18.810 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:18.810 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-22T06:37:19.047 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:20.137 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:19 smithi105 ceph-mon[27916]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.137 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:19 smithi105 ceph-mon[27916]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:20.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:19 smithi112 ceph-mon[31236]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:19 smithi112 ceph-mon[31236]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:19 smithi177 ceph-mon[31346]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:20.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:19 smithi177 ceph-mon[31346]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- 172.21.15.105:0/541085187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb81087a0 msgr2=0x7f5eb810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 --2- 172.21.15.105:0/541085187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb81087a0 0x7f5eb810ac10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000b0a0 tx=0x7f5eb000b3b0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- 172.21.15.105:0/541085187 shutdown_connections 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 --2- 172.21.15.105:0/541085187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb81087a0 0x7f5eb810ac10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 --2- 172.21.15.105:0/541085187 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb8103540 0x7f5eb8105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 --2- 172.21.15.105:0/541085187 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8105e70 0x7f5eb8108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- 172.21.15.105:0/541085187 >> 172.21.15.105:0/541085187 conn(0x7f5eb80fcf70 msgr2=0x7f5eb80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- 172.21.15.105:0/541085187 shutdown_connections 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- 172.21.15.105:0/541085187 wait complete. 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 Processor -- start 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.693+0000 7f5ec014e700 1 -- start start 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8103540 0x7f5eb8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb81087a0 0x7f5eb81a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb810e510 con 0x7f5eb8105e70 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb810e390 con 0x7f5eb8103540 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ec014e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb810e690 con 0x7f5eb81087a0 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51622/0 (socket says 172.21.15.105:51622) 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 -- 172.21.15.105:0/3273759096 learned_addr learned my addr 172.21.15.105:0/3273759096 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:20.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebdeea700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8103540 0x7f5eb8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebe6eb700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb81087a0 0x7f5eb81a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 -- 172.21.15.105:0/3273759096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb81087a0 msgr2=0x7f5eb81a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb81087a0 0x7f5eb81a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 -- 172.21.15.105:0/3273759096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8103540 msgr2=0x7f5eb8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8103540 0x7f5eb8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 -- 172.21.15.105:0/3273759096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb81a07c0 con 0x7f5eb8105e70 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5ebd6e9700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c9e0 tx=0x7f5ea800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8007ac0 con 0x7f5eb8105e70 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ea8007c60 con 0x7f5eb8105e70 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.694+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8018660 con 0x7f5eb8105e70 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.695+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb81a0ab0 con 0x7f5eb8105e70 2024-04-22T06:37:20.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.695+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb81a0fa0 con 0x7f5eb8105e70 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.696+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb8069730 con 0x7f5eb8105e70 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.697+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f5ea800f450 con 0x7f5eb8105e70 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.697+0000 7f5eaeffd700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 0x7f5ea40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.697+0000 7f5ebdeea700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 0x7f5ea40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.697+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5ea80d0340 con 0x7f5eb8105e70 2024-04-22T06:37:20.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.698+0000 7f5ebdeea700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 0x7f5ea40a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4005950 tx=0x7f5eb400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:20.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.702+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ea8014030 con 0x7f5eb8105e70 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.947+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f5eb802e190 con 0x7f5eb8105e70 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.947+0000 7f5eaeffd700 1 -- 172.21.15.105:0/3273759096 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3100 (secure 0 0 0) 0x7f5ea809cdc0 con 0x7f5eb8105e70 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:e26 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:legacy client fscid: 1 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:Filesystem 'cephfs' (1) 2024-04-22T06:37:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:fs_name cephfs 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 26 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-22T06:35:40.134334+0000 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:modified 2024-04-22T06:35:47.003690+0000 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:tableserver 0 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:root 0 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:session_timeout 300 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:session_autoclose 300 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:max_file_size 1099511627776 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:max_xattr_size 65536 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:required_client_features {} 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure 0 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:last_failure_osd_epoch 0 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-22T06:37:20.947 INFO:teuthology.orchestra.run.smithi105.stdout:max_mds 5 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:in 0,1,2,3,4 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:up {0=24500,1=24472,2=24461,3=24454,4=14640} 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:failed 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:damaged 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:stopped 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:data_pools [3] 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:metadata_pool 2 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:inline_data disabled 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:balancer 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:bal_rank_mask -1 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:standby_count_wanted 0 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:qdb_cluster leader: 24500 members: 24472,24500,24461,24454,14640 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.b{0:24500} state up:active seq 3 addr [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.948 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.f{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.177:6834/3449982583,v1:172.21.15.177:6835/3449982583] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.i{1:24472} state up:active seq 6 addr [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.e{1:24431} state up:standby-replay seq 1 addr [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.h{2:24461} state up:active seq 6 addr [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.k{2:24398} state up:standby-replay seq 1 addr [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.c{3:24454} state up:active seq 7 addr [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.l{3:24397} state up:standby-replay seq 1 addr [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.a{4:14640} state up:active seq 5 addr [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.g{4:14622} state up:standby-replay seq 1 addr [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:Standby daemons: 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.105:6834/3495232000,v1:172.21.15.105:6835/3495232000] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.949 INFO:teuthology.orchestra.run.smithi105.stdout:[mds.d{-1:14556} state up:standby seq 1 addr [v2:172.21.15.105:6836/2718881173,v1:172.21.15.105:6837/2718881173] compat {c=[1],r=[1],i=[1fff]}] 2024-04-22T06:37:20.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 msgr2=0x7f5ea40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:20.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 0x7f5ea40a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4005950 tx=0x7f5eb400a300 comp rx=0 tx=0).stop 2024-04-22T06:37:20.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 msgr2=0x7f5eb81990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800c9e0 tx=0x7f5ea800ccf0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 shutdown_connections 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5ea40a3c40 0x7f5ea40a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eb8103540 0x7f5eb8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5eb81087a0 0x7f5eb81a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 --2- 172.21.15.105:0/3273759096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5eb8105e70 0x7f5eb81990a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 >> 172.21.15.105:0/3273759096 conn(0x7f5eb80fcf70 msgr2=0x7f5eb8103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 shutdown_connections 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:20.950+0000 7f5ec014e700 1 -- 172.21.15.105:0/3273759096 wait complete. 2024-04-22T06:37:20.951 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:21.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-22T06:37:21.989 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:22.110 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:21 smithi105 ceph-mon[27916]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:22.111 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:21 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3273759096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:37:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:21 smithi112 ceph-mon[31236]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:21 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3273759096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:21 smithi177 ceph-mon[31346]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-22T06:37:22.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:21 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3273759096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 -- 172.21.15.105:0/4133509866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524103d90 msgr2=0x7f05241041f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 --2- 172.21.15.105:0/4133509866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524103d90 0x7f05241041f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0520009990 tx=0x7f0520009ca0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 -- 172.21.15.105:0/4133509866 shutdown_connections 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 --2- 172.21.15.105:0/4133509866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 0x7f05241011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 --2- 172.21.15.105:0/4133509866 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524104730 0x7f052410afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 --2- 172.21.15.105:0/4133509866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524103d90 0x7f05241041f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 -- 172.21.15.105:0/4133509866 >> 172.21.15.105:0/4133509866 conn(0x7f05240fc710 msgr2=0x7f05240feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 -- 172.21.15.105:0/4133509866 shutdown_connections 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.719+0000 7f052c5da700 1 -- 172.21.15.105:0/4133509866 wait complete. 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 Processor -- start 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 -- start start 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 0x7f05241036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 0x7f0524101d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f052410e550 con 0x7f0524104730 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f052410e3d0 con 0x7f0524100db0 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052c5da700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f052410e6d0 con 0x7f0524103d90 2024-04-22T06:37:23.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51634/0 (socket says 172.21.15.105:51634) 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 -- 172.21.15.105:0/3782145611 learned_addr learned my addr 172.21.15.105:0/3782145611 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052a376700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 0x7f05241036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f0529b75700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 0x7f0524101d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 -- 172.21.15.105:0/3782145611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 msgr2=0x7f0524101d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 0x7f0524101d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 -- 172.21.15.105:0/3782145611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 msgr2=0x7f05241036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.720+0000 7f052ab77700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 0x7f05241036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f052ab77700 1 -- 172.21.15.105:0/3782145611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0524102b30 con 0x7f0524104730 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f0529b75700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 0x7f0524101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f052ab77700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f051c007d70 tx=0x7f051c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c016020 con 0x7f0524104730 2024-04-22T06:37:23.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05241a2cc0 con 0x7f0524104730 2024-04-22T06:37:23.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f051c00ad40 con 0x7f0524104730 2024-04-22T06:37:23.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c0115e0 con 0x7f0524104730 2024-04-22T06:37:23.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.721+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05241a3180 con 0x7f0524104730 2024-04-22T06:37:23.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.722+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0524069730 con 0x7f0524104730 2024-04-22T06:37:23.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.724+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f051c019020 con 0x7f0524104730 2024-04-22T06:37:23.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.724+0000 7f051b7fe700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 0x7f05100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:23.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.724+0000 7f052a376700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 0x7f05100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:23.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.724+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f051c0d0870 con 0x7f0524104730 2024-04-22T06:37:23.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.725+0000 7f052a376700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 0x7f05100a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f052406af70 tx=0x7f051400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:23.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.728+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f051c020020 con 0x7f0524104730 2024-04-22T06:37:23.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.931+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f052402df80 con 0x7f0524104730 2024-04-22T06:37:23.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.932+0000 7f051b7fe700 1 -- 172.21.15.105:0/3782145611 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5255 (secure 0 0 0) 0x7f051c09d2f0 con 0x7f0524104730 2024-04-22T06:37:23.931 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 91 2024-04-22T06:37:23.931 INFO:teuthology.orchestra.run.smithi105.stdout:fsid 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:37:23.931 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-22T06:27:51.041097+0000 2024-04-22T06:37:23.931 INFO:teuthology.orchestra.run.smithi105.stdout:modified 2024-04-22T06:36:45.776076+0000 2024-04-22T06:37:23.931 INFO:teuthology.orchestra.run.smithi105.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:crush_version 26 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:full_ratio 0.95 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:backfillfull_ratio 0.9 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:nearfull_ratio 0.85 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:require_min_compat_client luminous 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:min_compat_client luminous 2024-04-22T06:37:23.932 INFO:teuthology.orchestra.run.smithi105.stdout:require_osd_release squid 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:stretch_mode_enabled false 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:max_osd 12 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6802/1950705877,v1:172.21.15.105:6803/1950705877] [v2:172.21.15.105:6804/1950705877,v1:172.21.15.105:6805/1950705877] exists,up 3a4d5ec0-d76f-4b44-9fcc-01b111b51a17 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/3644741495,v1:172.21.15.112:6801/3644741495] [v2:172.21.15.112:6802/3644741495,v1:172.21.15.112:6803/3644741495] exists,up e4ded2dc-2b79-4cd1-a34a-82778a1120b0 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1615473709,v1:172.21.15.177:6801/1615473709] [v2:172.21.15.177:6802/1615473709,v1:172.21.15.177:6803/1615473709] exists,up 9d48dd22-4dc5-4b82-bb01-119c1777318c 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6810/3727777768,v1:172.21.15.105:6811/3727777768] [v2:172.21.15.105:6812/3727777768,v1:172.21.15.105:6813/3727777768] exists,up b765227c-f85b-48e2-bc46-0de059180664 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 up in weight 1 up_from 34 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/2758455817,v1:172.21.15.112:6809/2758455817] [v2:172.21.15.112:6810/2758455817,v1:172.21.15.112:6811/2758455817] exists,up 1ae3793c-2262-486a-8754-ab46bcea415d 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 up in weight 1 up_from 40 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/1059606391,v1:172.21.15.177:6809/1059606391] [v2:172.21.15.177:6810/1059606391,v1:172.21.15.177:6811/1059606391] exists,up 6f9fd1a3-a0ef-4d21-afed-1aca4e529af0 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6 up in weight 1 up_from 48 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6818/546897953,v1:172.21.15.105:6819/546897953] [v2:172.21.15.105:6820/546897953,v1:172.21.15.105:6821/546897953] exists,up 2b6ab8f3-9288-43b7-886b-ce713b206fa1 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7 up in weight 1 up_from 53 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/1044643944,v1:172.21.15.112:6817/1044643944] [v2:172.21.15.112:6818/1044643944,v1:172.21.15.112:6819/1044643944] exists,up 1d324cb7-ac5a-44a5-b33f-4fb140559f77 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.8 up in weight 1 up_from 62 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/76921589,v1:172.21.15.177:6817/76921589] [v2:172.21.15.177:6818/76921589,v1:172.21.15.177:6819/76921589] exists,up 6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab 2024-04-22T06:37:23.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.9 up in weight 1 up_from 70 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6826/716531130,v1:172.21.15.105:6827/716531130] [v2:172.21.15.105:6828/716531130,v1:172.21.15.105:6829/716531130] exists,up 0ed862a3-bff9-49d5-abfe-ff784bbad0aa 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:osd.10 up in weight 1 up_from 76 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/2010680205,v1:172.21.15.112:6825/2010680205] [v2:172.21.15.112:6826/2010680205,v1:172.21.15.112:6827/2010680205] exists,up 8cd8511d-79e2-438b-8e64-d3936ea4fc3f 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:osd.11 up in weight 1 up_from 84 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/1875698423,v1:172.21.15.177:6825/1875698423] [v2:172.21.15.177:6826/1875698423,v1:172.21.15.177:6827/1875698423] exists,up 747a1047-9993-4f2b-9650-a2cddcbe04ff 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 2.13 [2,11] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 2.1d [8,11] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 3.1 [4,9] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 3.11 [5,9] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 3.14 [4,6] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:pg_upmap_items 3.34 [4,9] 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:6801/3871303088 expires 2024-04-23T06:28:44.539275+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/937171571 expires 2024-04-23T06:28:44.539275+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:6800/2332170576 expires 2024-04-23T06:28:17.391954+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/3406296347 expires 2024-04-23T06:28:44.539275+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:6800/3871303088 expires 2024-04-23T06:28:44.539275+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/2711052846 expires 2024-04-23T06:28:17.391954+0000 2024-04-22T06:37:23.934 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/1118894802 expires 2024-04-23T06:28:44.539275+0000 2024-04-22T06:37:23.935 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/1361172178 expires 2024-04-23T06:28:17.391954+0000 2024-04-22T06:37:23.935 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:6801/2332170576 expires 2024-04-23T06:28:17.391954+0000 2024-04-22T06:37:23.935 INFO:teuthology.orchestra.run.smithi105.stdout:blocklist 172.21.15.105:0/768203659 expires 2024-04-23T06:28:17.391954+0000 2024-04-22T06:37:23.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 msgr2=0x7f05100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:23.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 0x7f05100a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f052406af70 tx=0x7f051400a380 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 msgr2=0x7f0524102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f051c007d70 tx=0x7f051c00a340 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 shutdown_connections 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05100a3c40 0x7f05100a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0524100db0 0x7f05241036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0524104730 0x7f0524102280 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 --2- 172.21.15.105:0/3782145611 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0524103d90 0x7f0524101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.934+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 >> 172.21.15.105:0/3782145611 conn(0x7f05240fc710 msgr2=0x7f0524107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.935+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 shutdown_connections 2024-04-22T06:37:23.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:23.935+0000 7f052c5da700 1 -- 172.21.15.105:0/3782145611 wait complete. 2024-04-22T06:37:24.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:23 smithi105 ceph-mon[27916]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:24.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:23 smithi112 ceph-mon[31236]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:24.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:23 smithi177 ceph-mon[31346]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:24.693 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-22T06:37:24.705 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- 172.21.15.105:0/2038693775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214107d10 msgr2=0x7f021410a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 --2- 172.21.15.105:0/2038693775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214107d10 0x7f021410a100 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f020c009ca0 tx=0x7f020c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- 172.21.15.105:0/2038693775 shutdown_connections 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 --2- 172.21.15.105:0/2038693775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214107d10 0x7f021410a100 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 --2- 172.21.15.105:0/2038693775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02141053e0 0x7f02141077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 --2- 172.21.15.105:0/2038693775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0214102ab0 0x7f0214104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- 172.21.15.105:0/2038693775 >> 172.21.15.105:0/2038693775 conn(0x7f02140fc3e0 msgr2=0x7f02140fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- 172.21.15.105:0/2038693775 shutdown_connections 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- 172.21.15.105:0/2038693775 wait complete. 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 Processor -- start 2024-04-22T06:37:24.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.815+0000 7f021a474640 1 -- start start 2024-04-22T06:37:24.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:24.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 0x7f02141986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:24.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214107d10 0x7f021419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:24.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f021410dac0 con 0x7f0214102ab0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f021410d940 con 0x7f02141053e0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f021a474640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f021410dc40 con 0x7f0214107d10 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56044/0 (socket says 172.21.15.105:56044) 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 -- 172.21.15.105:0/819917301 learned_addr learned my addr 172.21.15.105:0/819917301 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f02137fe640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 0x7f02141986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0218a11640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214107d10 0x7f021419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 -- 172.21.15.105:0/819917301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214107d10 msgr2=0x7f021419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214107d10 0x7f021419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 -- 172.21.15.105:0/819917301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 msgr2=0x7f02141986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 0x7f02141986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 -- 172.21.15.105:0/819917301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021419fd90 con 0x7f0214102ab0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f02137fe640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 0x7f02141986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.816+0000 7f0213fff640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f020400e9f0 tx=0x7f020400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:24.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.817+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020400cd80 con 0x7f0214102ab0 2024-04-22T06:37:24.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.817+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02040108e0 con 0x7f0214102ab0 2024-04-22T06:37:24.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.817+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02141a0080 con 0x7f0214102ab0 2024-04-22T06:37:24.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.817+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204018980 con 0x7f0214102ab0 2024-04-22T06:37:24.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.817+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02141a0620 con 0x7f0214102ab0 2024-04-22T06:37:24.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.818+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02140fded0 con 0x7f0214102ab0 2024-04-22T06:37:24.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.819+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0204010430 con 0x7f0214102ab0 2024-04-22T06:37:24.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.820+0000 7f02117fa640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 0x7f01fc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:24.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.820+0000 7f02137fe640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 0x7f01fc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:24.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.820+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f02040d0620 con 0x7f0214102ab0 2024-04-22T06:37:24.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.820+0000 7f02137fe640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 0x7f01fc0a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0214199680 tx=0x7f0208005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:24.825+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f020409ca50 con 0x7f0214102ab0 2024-04-22T06:37:24.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:24 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3782145611' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:37:25.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.003+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0214062b50 con 0x7f0214102ab0 2024-04-22T06:37:25.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.005+0000 7f02117fa640 1 -- 172.21.15.105:0/819917301 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7f02040a0d50 con 0x7f0214102ab0 2024-04-22T06:37:25.003 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:25.003 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.006+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 msgr2=0x7f01fc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.006+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 0x7f01fc0a60f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0214199680 tx=0x7f0208005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.006+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 msgr2=0x7f0214198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.006+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f020400e9f0 tx=0x7f020400eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 shutdown_connections 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f01fc0a3c30 0x7f01fc0a60f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0214107d10 0x7f021419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02141053e0 0x7f02141986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 --2- 172.21.15.105:0/819917301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0214102ab0 0x7f0214198170 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 >> 172.21.15.105:0/819917301 conn(0x7f02140fc3e0 msgr2=0x7f0214103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 shutdown_connections 2024-04-22T06:37:25.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.007+0000 7f021a474640 1 -- 172.21.15.105:0/819917301 wait complete. 2024-04-22T06:37:25.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:25.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.120+0000 7fb8a7e81640 1 -- 172.21.15.105:0/834867962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a01029a0 msgr2=0x7fb8a01092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.120+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/834867962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a01029a0 0x7fb8a01092c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb898009ca0 tx=0x7fb89802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.120+0000 7fb8a7e81640 1 -- 172.21.15.105:0/834867962 shutdown_connections 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/834867962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a01029a0 0x7fb8a01092c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/834867962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a0101fe0 0x7fb8a0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/834867962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8a0100e90 0x7fb8a01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- 172.21.15.105:0/834867962 >> 172.21.15.105:0/834867962 conn(0x7fb8a00fc400 msgr2=0x7fb8a00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- 172.21.15.105:0/834867962 shutdown_connections 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- 172.21.15.105:0/834867962 wait complete. 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 Processor -- start 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- start start 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8a0100e90 0x7fb8a0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a01029a0 0x7fb8a019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a010ca40 con 0x7fb8a0101fe0 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a010c8c0 con 0x7fb8a01029a0 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a7e81640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a010cbc0 con 0x7fb8a0100e90 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a541c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a541c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56060/0 (socket says 172.21.15.105:56060) 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.121+0000 7fb8a541c640 1 -- 172.21.15.105:0/1276575027 learned_addr learned my addr 172.21.15.105:0/1276575027 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 -- 172.21.15.105:0/1276575027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8a0100e90 msgr2=0x7fb8a0198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8a0100e90 0x7fb8a0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 -- 172.21.15.105:0/1276575027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a01029a0 msgr2=0x7fb8a019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a641e640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a01029a0 0x7fb8a019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a01029a0 0x7fb8a019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 -- 172.21.15.105:0/1276575027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a019fe80 con 0x7fb8a0101fe0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a541c640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb89000c970 tx=0x7fb89000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890004220 con 0x7fb8a0101fe0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a01a0170 con 0x7fb8a0101fe0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8900043c0 con 0x7fb8a0101fe0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000f450 con 0x7fb8a0101fe0 2024-04-22T06:37:25.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.122+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a01a06c0 con 0x7fb8a0101fe0 2024-04-22T06:37:25.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.123+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a01a0980 con 0x7fb8a0101fe0 2024-04-22T06:37:25.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.124+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fb890003610 con 0x7fb8a0101fe0 2024-04-22T06:37:25.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.125+0000 7fb896ffd640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 0x7fb88c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.125+0000 7fb8a5c1d640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 0x7fb88c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.125+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fb8900d0290 con 0x7fb8a0101fe0 2024-04-22T06:37:25.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.125+0000 7fb8a5c1d640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 0x7fb88c0a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009710 tx=0x7fb89c006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.129+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb890014030 con 0x7fb8a0101fe0 2024-04-22T06:37:25.141 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:24 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3782145611' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:37:25.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:24 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3782145611' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-22T06:37:25.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.303+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8a01a0b50 con 0x7fb8a0101fe0 2024-04-22T06:37:25.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.303+0000 7fb896ffd640 1 -- 172.21.15.105:0/1276575027 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fb89009cd10 con 0x7fb8a0101fe0 2024-04-22T06:37:25.302 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:25.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 msgr2=0x7fb88c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 0x7fb88c0a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009710 tx=0x7fb89c006c90 comp rx=0 tx=0).stop 2024-04-22T06:37:25.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 msgr2=0x7fb8a01987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb89000c970 tx=0x7fb89000ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 shutdown_connections 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fb88c0a3bb0 0x7fb88c0a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8a01029a0 0x7fb8a019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8a0101fe0 0x7fb8a01987a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 --2- 172.21.15.105:0/1276575027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8a0100e90 0x7fb8a0198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 >> 172.21.15.105:0/1276575027 conn(0x7fb8a00fc400 msgr2=0x7fb8a00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 shutdown_connections 2024-04-22T06:37:25.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.305+0000 7fb8a7e81640 1 -- 172.21.15.105:0/1276575027 wait complete. 2024-04-22T06:37:25.315 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:25.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 -- 172.21.15.105:0/451367550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c101fe0 msgr2=0x7f820c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 --2- 172.21.15.105:0/451367550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c101fe0 0x7f820c102460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f81fc009900 tx=0x7f81fc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 -- 172.21.15.105:0/451367550 shutdown_connections 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 --2- 172.21.15.105:0/451367550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f820c1029a0 0x7f820c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 --2- 172.21.15.105:0/451367550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c101fe0 0x7f820c102460 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 --2- 172.21.15.105:0/451367550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 -- 172.21.15.105:0/451367550 >> 172.21.15.105:0/451367550 conn(0x7f820c0fc400 msgr2=0x7f820c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 -- 172.21.15.105:0/451367550 shutdown_connections 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.418+0000 7f82147b2640 1 -- 172.21.15.105:0/451367550 wait complete. 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 Processor -- start 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- start start 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f820c101fe0 0x7f820c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10db30 con 0x7f820c1029a0 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10d9b0 con 0x7f820c101fe0 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f820c10dcb0 con 0x7f820c100e90 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56082/0 (socket says 172.21.15.105:56082) 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 -- 172.21.15.105:0/2393109099 learned_addr learned my addr 172.21.15.105:0/2393109099 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f821254e640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 -- 172.21.15.105:0/2393109099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 msgr2=0x7f820c198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 -- 172.21.15.105:0/2393109099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f820c101fe0 msgr2=0x7f820c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f820c101fe0 0x7f820c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 -- 172.21.15.105:0/2393109099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c19fe90 con 0x7f820c1029a0 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f821254e640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:25.420 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f8212d4f640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f820400cad0 tx=0x7f820400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820400ae20 con 0x7f820c1029a0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8204005020 con 0x7f820c1029a0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820400f6b0 con 0x7f820c1029a0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c1a0180 con 0x7f820c1029a0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.419+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c1a0680 con 0x7f820c1029a0 2024-04-22T06:37:25.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.420+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c108330 con 0x7f820c1029a0 2024-04-22T06:37:25.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.421+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f82040051d0 con 0x7f820c1029a0 2024-04-22T06:37:25.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.422+0000 7f82037fe640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 0x7f81f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.422+0000 7f821254e640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 0x7f81f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.422+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f82040d04d0 con 0x7f820c1029a0 2024-04-22T06:37:25.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.422+0000 7f821254e640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 0x7f81f80a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8208009b40 tx=0x7f8208005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.427+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8204014030 con 0x7f820c1029a0 2024-04-22T06:37:25.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.605+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f820c02d810 con 0x7f820c1029a0 2024-04-22T06:37:25.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.605+0000 7f82037fe640 1 -- 172.21.15.105:0/2393109099 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f820409cf50 con 0x7f820c1029a0 2024-04-22T06:37:25.604 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 msgr2=0x7f81f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 0x7f81f80a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8208009b40 tx=0x7f8208005e50 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 msgr2=0x7f820c19f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f820400cad0 tx=0x7f820400cf90 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 shutdown_connections 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f81f80a3bb0 0x7f81f80a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f820c1029a0 0x7f820c19f820 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f820c101fe0 0x7f820c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 --2- 172.21.15.105:0/2393109099 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f820c100e90 0x7f820c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 >> 172.21.15.105:0/2393109099 conn(0x7f820c0fc400 msgr2=0x7f820c105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 shutdown_connections 2024-04-22T06:37:25.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.607+0000 7f82147b2640 1 -- 172.21.15.105:0/2393109099 wait complete. 2024-04-22T06:37:25.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- 172.21.15.105:0/1731003614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4103230 msgr2=0x7f3ba41036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 --2- 172.21.15.105:0/1731003614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4103230 0x7f3ba41036b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009930 tx=0x7f3b9002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- 172.21.15.105:0/1731003614 shutdown_connections 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 --2- 172.21.15.105:0/1731003614 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 0x7f3ba410a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 --2- 172.21.15.105:0/1731003614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4103230 0x7f3ba41036b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 --2- 172.21.15.105:0/1731003614 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4100260 0x7f3ba4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- 172.21.15.105:0/1731003614 >> 172.21.15.105:0/1731003614 conn(0x7f3ba40fbb90 msgr2=0x7f3ba40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- 172.21.15.105:0/1731003614 shutdown_connections 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- 172.21.15.105:0/1731003614 wait complete. 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 Processor -- start 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.719+0000 7f3baab6d640 1 -- start start 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4103230 0x7f3ba41987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 0x7f3ba419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba410db00 con 0x7f3ba4100260 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba410d980 con 0x7f3ba4103bf0 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba410dc80 con 0x7f3ba4103230 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56098/0 (socket says 172.21.15.105:56098) 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 -- 172.21.15.105:0/2074525664 learned_addr learned my addr 172.21.15.105:0/2074525664 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba910a640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 0x7f3ba419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba3fff640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4103230 0x7f3ba41987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 -- 172.21.15.105:0/2074525664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4103230 msgr2=0x7f3ba41987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4103230 0x7f3ba41987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 -- 172.21.15.105:0/2074525664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 msgr2=0x7f3ba419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 0x7f3ba419f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 -- 172.21.15.105:0/2074525664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba419fe80 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba8909640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c970 tx=0x7f3b9800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004220 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba41a0170 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b980043c0 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004a00 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.720+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba41a0670 con 0x7f3ba4100260 2024-04-22T06:37:25.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.721+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba4109480 con 0x7f3ba4100260 2024-04-22T06:37:25.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.722+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f3b980079b0 con 0x7f3ba4100260 2024-04-22T06:37:25.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.723+0000 7f3ba1ffb640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 0x7f3b940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:25.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.723+0000 7f3ba3fff640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 0x7f3b940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:25.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.723+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f3b980d0170 con 0x7f3ba4100260 2024-04-22T06:37:25.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.724+0000 7f3ba3fff640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 0x7f3b940a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002c840 tx=0x7f3b90000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:25.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.728+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b98014030 con 0x7f3ba4100260 2024-04-22T06:37:25.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.907+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ba4062b50 con 0x7f3ba4100260 2024-04-22T06:37:25.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.908+0000 7f3ba1ffb640 1 -- 172.21.15.105:0/2074525664 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f3b9809cbf0 con 0x7f3ba4100260 2024-04-22T06:37:25.907 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:25.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.909+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 msgr2=0x7f3b940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.909+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 0x7f3b940a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002c840 tx=0x7f3b90000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 msgr2=0x7f3ba4198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c970 tx=0x7f3b9800ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 shutdown_connections 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f3b940a3bb0 0x7f3b940a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ba4103bf0 0x7f3ba419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ba4103230 0x7f3ba41987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 --2- 172.21.15.105:0/2074525664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ba4100260 0x7f3ba4198260 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 >> 172.21.15.105:0/2074525664 conn(0x7f3ba40fbb90 msgr2=0x7f3ba4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 shutdown_connections 2024-04-22T06:37:25.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:25.910+0000 7f3baab6d640 1 -- 172.21.15.105:0/2074525664 wait complete. 2024-04-22T06:37:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:25 smithi105 ceph-mon[27916]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/819917301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1276575027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2393109099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:25.919 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:26.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 -- 172.21.15.105:0/3349429192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23301011a0 msgr2=0x7f2330101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 --2- 172.21.15.105:0/3349429192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23301011a0 0x7f2330101600 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2324009900 tx=0x7f232402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:26.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 -- 172.21.15.105:0/3349429192 shutdown_connections 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 --2- 172.21.15.105:0/3349429192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330101b40 0x7f2330101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 --2- 172.21.15.105:0/3349429192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23301011a0 0x7f2330101600 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 --2- 172.21.15.105:0/3349429192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2330100840 0x7f2330100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 -- 172.21.15.105:0/3349429192 >> 172.21.15.105:0/3349429192 conn(0x7f23300fc3e0 msgr2=0x7f23300fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 -- 172.21.15.105:0/3349429192 shutdown_connections 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 -- 172.21.15.105:0/3349429192 wait complete. 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.018+0000 7f233599d640 1 Processor -- start 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 -- start start 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 0x7f233019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 0x7f233019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f233010fc90 con 0x7f2330101b40 2024-04-22T06:37:26.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f233010fb10 con 0x7f23301011a0 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f233010fe10 con 0x7f2330100840 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56110/0 (socket says 172.21.15.105:56110) 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 -- 172.21.15.105:0/3272322365 learned_addr learned my addr 172.21.15.105:0/3272322365 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232e7fc640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 0x7f233019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232effd640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 0x7f233019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 -- 172.21.15.105:0/3272322365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 msgr2=0x7f233019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 0x7f233019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 -- 172.21.15.105:0/3272322365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 msgr2=0x7f233019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 0x7f233019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 -- 172.21.15.105:0/3272322365 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23301a1fe0 con 0x7f2330101b40 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232e7fc640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 0x7f233019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232effd640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 0x7f233019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f232f7fe640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f232800cad0 tx=0x7f232800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800ae20 con 0x7f2330101b40 2024-04-22T06:37:26.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.019+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23301a22d0 con 0x7f2330101b40 2024-04-22T06:37:26.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.020+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2328005020 con 0x7f2330101b40 2024-04-22T06:37:26.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.020+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800f6b0 con 0x7f2330101b40 2024-04-22T06:37:26.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.020+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23301a2820 con 0x7f2330101b40 2024-04-22T06:37:26.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.021+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233010b810 con 0x7f2330101b40 2024-04-22T06:37:26.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.022+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f23280051d0 con 0x7f2330101b40 2024-04-22T06:37:26.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.023+0000 7f233499b640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.023+0000 7f232effd640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.023+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f23280d0900 con 0x7f2330101b40 2024-04-22T06:37:26.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.023+0000 7f232effd640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 0x7f23180a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f23200058c0 tx=0x7f2320005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.027+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2328014030 con 0x7f2330101b40 2024-04-22T06:37:26.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:25 smithi112 ceph-mon[31236]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:26.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/819917301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:26.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1276575027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:26.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2393109099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:26.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:25 smithi177 ceph-mon[31346]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/819917301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1276575027' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:26.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2393109099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:26.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.244+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f233002d5c0 con 0x7f2330101b40 2024-04-22T06:37:26.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.245+0000 7f233499b640 1 -- 172.21.15.105:0/3272322365 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7f232809d2e0 con 0x7f2330101b40 2024-04-22T06:37:26.243 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:26.243 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:37:26.243 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:26.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.246+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 msgr2=0x7f23180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.246+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 0x7f23180a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f23200058c0 tx=0x7f2320005d30 comp rx=0 tx=0).stop 2024-04-22T06:37:26.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.246+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 msgr2=0x7f23301a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.246+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f232800cad0 tx=0x7f232800cf90 comp rx=0 tx=0).stop 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 shutdown_connections 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2330101b40 0x7f23301a1970 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f23301011a0 0x7f233019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 --2- 172.21.15.105:0/3272322365 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2330100840 0x7f233019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 >> 172.21.15.105:0/3272322365 conn(0x7f23300fc3e0 msgr2=0x7f23300fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 shutdown_connections 2024-04-22T06:37:26.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.247+0000 7f233599d640 1 -- 172.21.15.105:0/3272322365 wait complete. 2024-04-22T06:37:26.257 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- 172.21.15.105:0/4220144046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 msgr2=0x7f66fc1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- 172.21.15.105:0/4220144046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc1092b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009ca0 tx=0x7f66f402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- 172.21.15.105:0/4220144046 shutdown_connections 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- 172.21.15.105:0/4220144046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc1092b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- 172.21.15.105:0/4220144046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc101fd0 0x7f66fc102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- 172.21.15.105:0/4220144046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc100e80 0x7f66fc1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- 172.21.15.105:0/4220144046 >> 172.21.15.105:0/4220144046 conn(0x7f66fc0fc410 msgr2=0x7f66fc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- 172.21.15.105:0/4220144046 shutdown_connections 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- 172.21.15.105:0/4220144046 wait complete. 2024-04-22T06:37:26.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 Processor -- start 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 -- start start 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc100e80 0x7f66fc197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.357+0000 7f67033c7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 0x7f66fc1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10ca50 con 0x7f66fc102990 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10c8d0 con 0x7f66fc101fd0 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10cbd0 con 0x7f66fc100e80 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56134/0 (socket says 172.21.15.105:56134) 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 -- 172.21.15.105:0/543745635 learned_addr learned my addr 172.21.15.105:0/543745635 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6700962640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 0x7f66fc1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701163640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc100e80 0x7f66fc197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 -- 172.21.15.105:0/543745635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc100e80 msgr2=0x7f66fc197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc100e80 0x7f66fc197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 -- 172.21.15.105:0/543745635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 msgr2=0x7f66fc1975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 0x7f66fc1975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 -- 172.21.15.105:0/543745635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc19ecb0 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6700962640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 0x7f66fc1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f6701964640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f66f402c700 tx=0x7f66f4034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4039040 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66fc19ef40 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66f4044070 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f40042a0 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.358+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66fc19f400 con 0x7f66fc102990 2024-04-22T06:37:26.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.359+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66fc108190 con 0x7f66fc102990 2024-04-22T06:37:26.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.360+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f66f4003970 con 0x7f66fc102990 2024-04-22T06:37:26.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.361+0000 7f66f27fc640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 0x7f66e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.361+0000 7f6701163640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 0x7f66e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.361+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f66f40f3320 con 0x7f66fc102990 2024-04-22T06:37:26.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.361+0000 7f6701163640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 0x7f66e80a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009890 tx=0x7f66f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.365+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f66f40bfda0 con 0x7f66fc102990 2024-04-22T06:37:26.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.546+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66fc02d7c0 con 0x7f66fc102990 2024-04-22T06:37:26.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.547+0000 7f66f27fc640 1 -- 172.21.15.105:0/543745635 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7f66f40bf6a0 con 0x7f66fc102990 2024-04-22T06:37:26.546 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:26.546 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 msgr2=0x7f66e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 0x7f66e80a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009890 tx=0x7f66f8008040 comp rx=0 tx=0).stop 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 msgr2=0x7f66fc19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f66f402c700 tx=0x7f66f4034760 comp rx=0 tx=0).stop 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 shutdown_connections 2024-04-22T06:37:26.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f66e80a3bb0 0x7f66e80a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66fc102990 0x7f66fc19e640 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.549+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66fc101fd0 0x7f66fc1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.550+0000 7f67033c7640 1 --2- 172.21.15.105:0/543745635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66fc100e80 0x7f66fc197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.550+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 >> 172.21.15.105:0/543745635 conn(0x7f66fc0fc410 msgr2=0x7f66fc105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.550+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 shutdown_connections 2024-04-22T06:37:26.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.550+0000 7f67033c7640 1 -- 172.21.15.105:0/543745635 wait complete. 2024-04-22T06:37:26.559 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:26.560 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/4121590623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04101fe0 msgr2=0x7f4f04102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/4121590623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04101fe0 0x7f4f04102460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009900 tx=0x7f4ef402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/4121590623 shutdown_connections 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/4121590623 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f041029a0 0x7f4f041092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/4121590623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04101fe0 0x7f4f04102460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/4121590623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f04100e90 0x7f4f041012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.661+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/4121590623 >> 172.21.15.105:0/4121590623 conn(0x7f4f040fc400 msgr2=0x7f4f040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/4121590623 shutdown_connections 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/4121590623 wait complete. 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 Processor -- start 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- start start 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f04101fe0 0x7f4f04198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f041029a0 0x7f4f0419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0410db30 con 0x7f4f04100e90 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0410d9b0 con 0x7f4f041029a0 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f0c1d5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0410dcb0 con 0x7f4f04101fe0 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f09f71640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f09f71640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56150/0 (socket says 172.21.15.105:56150) 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.662+0000 7f4f09f71640 1 -- 172.21.15.105:0/1246998362 learned_addr learned my addr 172.21.15.105:0/1246998362 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:26.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09770640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f04101fe0 0x7f4f04198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 -- 172.21.15.105:0/1246998362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f04101fe0 msgr2=0x7f4f04198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f04101fe0 0x7f4f04198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f0a772640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f041029a0 0x7f4f0419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 -- 172.21.15.105:0/1246998362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f041029a0 msgr2=0x7f4f0419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f041029a0 0x7f4f0419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 -- 172.21.15.105:0/1246998362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0419fe40 con 0x7f4f04100e90 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f09f71640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c890 tx=0x7f4f0000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00004220 con 0x7f4f04100e90 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f041a0130 con 0x7f4f04100e90 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f000043c0 con 0x7f4f04100e90 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00004a00 con 0x7f4f04100e90 2024-04-22T06:37:26.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.663+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f041a06a0 con 0x7f4f04100e90 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.664+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f04108270 con 0x7f4f04100e90 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.665+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f4f000079b0 con 0x7f4f04100e90 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.666+0000 7f4efaffd640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 0x7f4ef00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.666+0000 7f4f09770640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 0x7f4ef00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.666+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f4f000d0150 con 0x7f4f04100e90 2024-04-22T06:37:26.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.666+0000 7f4f09770640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 0x7f4ef00a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4006010 tx=0x7f4ef4005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.670+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f00014030 con 0x7f4f04100e90 2024-04-22T06:37:26.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.851+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f04062b50 con 0x7f4f04100e90 2024-04-22T06:37:26.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.851+0000 7f4efaffd640 1 -- 172.21.15.105:0/1246998362 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f4f0009cbd0 con 0x7f4f04100e90 2024-04-22T06:37:26.849 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 msgr2=0x7f4ef00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 0x7f4ef00a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4006010 tx=0x7f4ef4005be0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 msgr2=0x7f4f04198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c890 tx=0x7f4f0000cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 shutdown_connections 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f4ef00a3bb0 0x7f4ef00a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4f041029a0 0x7f4f0419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4f04101fe0 0x7f4f04198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 --2- 172.21.15.105:0/1246998362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f04100e90 0x7f4f04198220 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 >> 172.21.15.105:0/1246998362 conn(0x7f4f040fc400 msgr2=0x7f4f04105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 shutdown_connections 2024-04-22T06:37:26.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.853+0000 7f4f0c1d5640 1 -- 172.21.15.105:0/1246998362 wait complete. 2024-04-22T06:37:26.862 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2074525664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:26.862 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3272322365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:26.862 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/543745635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:26.863 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:26.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 -- 172.21.15.105:0/3097861730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360101fa0 msgr2=0x7f53601023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 --2- 172.21.15.105:0/3097861730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360101fa0 0x7f53601023c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5354009900 tx=0x7f535402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:26.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 -- 172.21.15.105:0/3097861730 shutdown_connections 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 --2- 172.21.15.105:0/3097861730 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360103b50 0x7f536010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 --2- 172.21.15.105:0/3097861730 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103190 0x7f5360103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 --2- 172.21.15.105:0/3097861730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360101fa0 0x7f53601023c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 -- 172.21.15.105:0/3097861730 >> 172.21.15.105:0/3097861730 conn(0x7f53600fd530 msgr2=0x7f53600ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.966+0000 7f5365a11640 1 -- 172.21.15.105:0/3097861730 shutdown_connections 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 -- 172.21.15.105:0/3097861730 wait complete. 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 Processor -- start 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 -- start start 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 0x7f5360198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 0x7f536019f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f536010dba0 con 0x7f5360103190 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f536010da20 con 0x7f5360101fa0 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f5365a11640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f536010dd20 con 0x7f5360103b50 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f535e7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f535e7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56160/0 (socket says 172.21.15.105:56160) 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.967+0000 7f535e7fc640 1 -- 172.21.15.105:0/2744984899 learned_addr learned my addr 172.21.15.105:0/2744984899 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535effd640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 0x7f5360198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535f7fe640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 0x7f536019f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 -- 172.21.15.105:0/2744984899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 msgr2=0x7f536019f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 0x7f536019f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 -- 172.21.15.105:0/2744984899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 msgr2=0x7f5360198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 0x7f5360198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 -- 172.21.15.105:0/2744984899 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536019ffb0 con 0x7f5360103190 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535effd640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 0x7f5360198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535f7fe640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 0x7f536019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:26.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f535e7fc640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f535000b4a0 tx=0x7f535000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350014070 con 0x7f5360103190 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53601a02a0 con 0x7f5360103190 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5350010ac0 con 0x7f5360103190 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350017c30 con 0x7f5360103190 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.968+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53601a0820 con 0x7f5360103190 2024-04-22T06:37:26.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.969+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5360109480 con 0x7f5360103190 2024-04-22T06:37:26.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.970+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f5350017070 con 0x7f5360103190 2024-04-22T06:37:26.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.971+0000 7f5364a0f640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 0x7f53480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:26.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.971+0000 7f535effd640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 0x7f53480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:26.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.971+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f53500cfba0 con 0x7f5360103190 2024-04-22T06:37:26.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.971+0000 7f535effd640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 0x7f53480a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5354006010 tx=0x7f53540095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:26.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:26.974+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f535009c620 con 0x7f5360103190 2024-04-22T06:37:27.110 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2074525664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:27.110 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3272322365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:27.110 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/543745635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:27.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.154+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f536002d810 con 0x7f5360103190 2024-04-22T06:37:27.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.154+0000 7f5364a0f640 1 -- 172.21.15.105:0/2744984899 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f535009bf20 con 0x7f5360103190 2024-04-22T06:37:27.153 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 msgr2=0x7f53480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 0x7f53480a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5354006010 tx=0x7f53540095f0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 msgr2=0x7f53601988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f535000b4a0 tx=0x7f535000b960 comp rx=0 tx=0).stop 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 shutdown_connections 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f53480a3bb0 0x7f53480a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5360103b50 0x7f536019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5360103190 0x7f53601988d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.156+0000 7f5365a11640 1 --2- 172.21.15.105:0/2744984899 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5360101fa0 0x7f5360198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.157+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 >> 172.21.15.105:0/2744984899 conn(0x7f53600fd530 msgr2=0x7f5360106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.157+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 shutdown_connections 2024-04-22T06:37:27.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.157+0000 7f5365a11640 1 -- 172.21.15.105:0/2744984899 wait complete. 2024-04-22T06:37:27.166 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:27.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2074525664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:27.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3272322365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:27.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/543745635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:27.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.268+0000 7f84a006a640 1 -- 172.21.15.105:0/2378926966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84981029a0 msgr2=0x7f84981092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.268+0000 7f84a006a640 1 --2- 172.21.15.105:0/2378926966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84981029a0 0x7f84981092c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8488009900 tx=0x7f848802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:27.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- 172.21.15.105:0/2378926966 shutdown_connections 2024-04-22T06:37:27.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- 172.21.15.105:0/2378926966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84981029a0 0x7f84981092c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- 172.21.15.105:0/2378926966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498101fe0 0x7f8498102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- 172.21.15.105:0/2378926966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8498100e90 0x7f84981012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- 172.21.15.105:0/2378926966 >> 172.21.15.105:0/2378926966 conn(0x7f84980fc400 msgr2=0x7f84980fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- 172.21.15.105:0/2378926966 shutdown_connections 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- 172.21.15.105:0/2378926966 wait complete. 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 Processor -- start 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- start start 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498100e90 0x7f84981981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 0x7f849819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f849810c9e0 con 0x7f8498101fe0 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f849810c860 con 0x7f84981029a0 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f84a006a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f849810cb60 con 0x7f8498100e90 2024-04-22T06:37:27.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f849d605640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f849d605640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56168/0 (socket says 172.21.15.105:56168) 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.269+0000 7f849d605640 1 -- 172.21.15.105:0/2863982133 learned_addr learned my addr 172.21.15.105:0/2863982133 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849de06640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498100e90 0x7f84981981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849e607640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 0x7f849819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 -- 172.21.15.105:0/2863982133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498100e90 msgr2=0x7f84981981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498100e90 0x7f84981981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 -- 172.21.15.105:0/2863982133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 msgr2=0x7f849819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 0x7f849819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 -- 172.21.15.105:0/2863982133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849819fe00 con 0x7f8498101fe0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849e607640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 0x7f849819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f849d605640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f848c00d950 tx=0x7f848c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c017070 con 0x7f8498101fe0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84981a00f0 con 0x7f8498101fe0 2024-04-22T06:37:27.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f848c009c30 con 0x7f8498101fe0 2024-04-22T06:37:27.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c00f620 con 0x7f8498101fe0 2024-04-22T06:37:27.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.270+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84981a0640 con 0x7f8498101fe0 2024-04-22T06:37:27.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.271+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84981a0960 con 0x7f8498101fe0 2024-04-22T06:37:27.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.272+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f848c01a070 con 0x7f8498101fe0 2024-04-22T06:37:27.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.273+0000 7f8496ffd640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 0x7f84800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.273+0000 7f849de06640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 0x7f84800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.273+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f848c0cfdf0 con 0x7f8498101fe0 2024-04-22T06:37:27.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.273+0000 7f849de06640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 0x7f84800a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8490005c40 tx=0x7f849000a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.277+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f848c009de0 con 0x7f8498101fe0 2024-04-22T06:37:27.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.456+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f84981a0b30 con 0x7f8498101fe0 2024-04-22T06:37:27.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.456+0000 7f8496ffd640 1 -- 172.21.15.105:0/2863982133 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f848c09c870 con 0x7f8498101fe0 2024-04-22T06:37:27.455 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 msgr2=0x7f84800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 0x7f84800a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8490005c40 tx=0x7f849000a590 comp rx=0 tx=0).stop 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 msgr2=0x7f8498198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f848c00d950 tx=0x7f848c00de10 comp rx=0 tx=0).stop 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 shutdown_connections 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f84800a3bb0 0x7f84800a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84981029a0 0x7f849819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8498101fe0 0x7f8498198720 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 --2- 172.21.15.105:0/2863982133 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8498100e90 0x7f84981981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 >> 172.21.15.105:0/2863982133 conn(0x7f84980fc400 msgr2=0x7f84980fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.458+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 shutdown_connections 2024-04-22T06:37:27.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.459+0000 7f84a006a640 1 -- 172.21.15.105:0/2863982133 wait complete. 2024-04-22T06:37:27.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 -- 172.21.15.105:0/2748920242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc101ff0 msgr2=0x7fa5fc102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 --2- 172.21.15.105:0/2748920242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc102470 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009900 tx=0x7fa5f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 -- 172.21.15.105:0/2748920242 shutdown_connections 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 --2- 172.21.15.105:0/2748920242 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 --2- 172.21.15.105:0/2748920242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc102470 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 --2- 172.21.15.105:0/2748920242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 -- 172.21.15.105:0/2748920242 >> 172.21.15.105:0/2748920242 conn(0x7fa5fc0fc410 msgr2=0x7fa5fc0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 -- 172.21.15.105:0/2748920242 shutdown_connections 2024-04-22T06:37:27.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.568+0000 7fa600a93640 1 -- 172.21.15.105:0/2748920242 wait complete. 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 Processor -- start 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 -- start start 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc06b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc10c9d0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc10c850 con 0x7fa5fc100ea0 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa600a93640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc10cb50 con 0x7fa5fc101ff0 2024-04-22T06:37:27.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56196/0 (socket says 172.21.15.105:56196) 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 -- 172.21.15.105:0/1885582440 learned_addr learned my addr 172.21.15.105:0/1885582440 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fa59c640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5f9d9b640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 -- 172.21.15.105:0/1885582440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc101ff0 msgr2=0x7fa5fc19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc19ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 -- 172.21.15.105:0/1885582440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 msgr2=0x7fa5fc06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fad9d640 1 -- 172.21.15.105:0/1885582440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc19fdd0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.569+0000 7fa5fa59c640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc06b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa5fad9d640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400eaa0 tx=0x7fa5f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400cdb0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f4005020 con 0x7fa5fc1029b0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc1a00c0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f40116d0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.570+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc1a0610 con 0x7fa5fc1029b0 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.571+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5fc108230 con 0x7fa5fc1029b0 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.572+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fa5f40051d0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.572+0000 7fa5eb7fe640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 0x7fa5e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.572+0000 7fa5fa59c640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 0x7fa5e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.572+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fa5f40d05d0 con 0x7fa5fc1029b0 2024-04-22T06:37:27.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.573+0000 7fa5fa59c640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 0x7fa5e40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0058c0 tx=0x7fa5ec005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.577+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5f4016030 con 0x7fa5fc1029b0 2024-04-22T06:37:27.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.794+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5fc02d770 con 0x7fa5fc1029b0 2024-04-22T06:37:27.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.795+0000 7fa5eb7fe640 1 -- 172.21.15.105:0/1885582440 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11947 (secure 0 0 0) 0x7fa5f409d050 con 0x7fa5fc1029b0 2024-04-22T06:37:27.793 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:35:47.003690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24472,24500,24461,24454,14640]},"id":1}]} 2024-04-22T06:37:27.794 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 26 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 msgr2=0x7fa5e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 0x7fa5e40a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0058c0 tx=0x7fa5ec005ec0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 msgr2=0x7fa5fc19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400eaa0 tx=0x7fa5f400ef60 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 shutdown_connections 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fa5e40a3bb0 0x7fa5e40a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5fc1029b0 0x7fa5fc19f510 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5fc101ff0 0x7fa5fc19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 --2- 172.21.15.105:0/1885582440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5fc100ea0 0x7fa5fc06b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 >> 172.21.15.105:0/1885582440 conn(0x7fa5fc0fc410 msgr2=0x7fa5fc105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 shutdown_connections 2024-04-22T06:37:27.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.797+0000 7fa600a93640 1 -- 172.21.15.105:0/1885582440 wait complete. 2024-04-22T06:37:27.807 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 -- 172.21.15.105:0/1952111470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 msgr2=0x7f1e9c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 --2- 172.21.15.105:0/1952111470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c1092a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009ca0 tx=0x7f1e9402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 -- 172.21.15.105:0/1952111470 shutdown_connections 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 --2- 172.21.15.105:0/1952111470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c1092a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 --2- 172.21.15.105:0/1952111470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 0x7f1e9c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 --2- 172.21.15.105:0/1952111470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 -- 172.21.15.105:0/1952111470 >> 172.21.15.105:0/1952111470 conn(0x7f1e9c0fc3e0 msgr2=0x7f1e9c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 -- 172.21.15.105:0/1952111470 shutdown_connections 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.906+0000 7f1ea1375640 1 -- 172.21.15.105:0/1952111470 wait complete. 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 Processor -- start 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 -- start start 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 0x7f1e9c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10c9a0 con 0x7f1e9c102980 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10c820 con 0x7f1e9c101fc0 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1ea1375640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c10cb20 con 0x7f1e9c100e70 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56226/0 (socket says 172.21.15.105:56226) 2024-04-22T06:37:27.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9b7fe640 1 -- 172.21.15.105:0/4114342686 learned_addr learned my addr 172.21.15.105:0/4114342686 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9a7fc640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 0x7f1e9c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9affd640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9b7fe640 1 -- 172.21.15.105:0/4114342686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 msgr2=0x7f1e9c196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.907+0000 7f1e9b7fe640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e9b7fe640 1 -- 172.21.15.105:0/4114342686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 msgr2=0x7f1e9c1974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e9b7fe640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 0x7f1e9c1974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e9b7fe640 1 -- 172.21.15.105:0/4114342686 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c19ebd0 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e9affd640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e9b7fe640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009dc0 tx=0x7f1e94032fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94034aa0 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c0690e0 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e94034c40 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940042a0 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.908+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c069620 con 0x7f1e9c102980 2024-04-22T06:37:27.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.909+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9c068de0 con 0x7f1e9c102980 2024-04-22T06:37:27.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.910+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f1e940325c0 con 0x7f1e9c102980 2024-04-22T06:37:27.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.910+0000 7f1e83fff640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 0x7f1e840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:27.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.911+0000 7f1e9affd640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 0x7f1e840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:27.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.911+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f1e940f4300 con 0x7f1e9c102980 2024-04-22T06:37:27.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.911+0000 7f1e9affd640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 0x7f1e840a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00baa0 tx=0x7f1e8c005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:27.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:27.914+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e940c0d80 con 0x7f1e9c102980 2024-04-22T06:37:28.092 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.094+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e9c107d20 con 0x7f1e9c102980 2024-04-22T06:37:28.094 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.095+0000 7f1e83fff640 1 -- 172.21.15.105:0/4114342686 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22606 (secure 0 0 0) 0x7f1e940c0680 con 0x7f1e9c102980 2024-04-22T06:37:28.094 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:28.094 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":91,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:36:45.776076+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:28.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.097+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 msgr2=0x7f1e840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.097+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 0x7f1e840a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00baa0 tx=0x7f1e8c005d90 comp rx=0 tx=0).stop 2024-04-22T06:37:28.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.097+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 msgr2=0x7f1e9c19e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.096 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.097+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009dc0 tx=0x7f1e94032fb0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 shutdown_connections 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f1e840a3bb0 0x7f1e840a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1e9c102980 0x7f1e9c19e560 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1e9c101fc0 0x7f1e9c1974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 --2- 172.21.15.105:0/4114342686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1e9c100e70 0x7f1e9c196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 >> 172.21.15.105:0/4114342686 conn(0x7f1e9c0fc3e0 msgr2=0x7f1e9c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 shutdown_connections 2024-04-22T06:37:28.097 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.098+0000 7f1ea1375640 1 -- 172.21.15.105:0/4114342686 wait complete. 2024-04-22T06:37:28.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:27 smithi105 ceph-mon[27916]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:28.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:27 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1246998362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:27 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2744984899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.106 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:27 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2863982133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:35:40.134334+0000', 'modified': '2024-04-22T06:35:47.003690+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24472, 'mds_2': 24461, 'mds_3': 24454, 'mds_4': 14640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6839/256609249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 256609249}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 256609249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14640': {'gid': 14640, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6841/2147123680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6840', 'nonce': 2147123680}, {'type': 'v1', 'addr': '172.21.15.105:6841', 'nonce': 2147123680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24397': {'gid': 24397, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6833/624977252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 624977252}, {'type': 'v1', 'addr': '172.21.15.177:6833', 'nonce': 624977252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/583171810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 583171810}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 583171810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/3883368202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 3883368202}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 3883368202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/3449982583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 3449982583}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 3449982583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.177:6837/4114728122', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 4114728122}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 4114728122}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6837/3970644798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3970644798}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3970644798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.177:6839/1381266768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 1381266768}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 1381266768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/4112363218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 4112363218}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 4112363218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24472, 24500, 24461, 24454, 14640]} 2024-04-22T06:37:28.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T06:37:28.108 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T06:37:28.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.208+0000 7f42f4775640 1 -- 172.21.15.105:0/1346521780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 msgr2=0x7f42ec104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.208+0000 7f42f4775640 1 --2- 172.21.15.105:0/1346521780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec104e80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009900 tx=0x7f42dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:28.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- 172.21.15.105:0/1346521780 shutdown_connections 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- 172.21.15.105:0/1346521780 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- 172.21.15.105:0/1346521780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec1077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- 172.21.15.105:0/1346521780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec104e80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- 172.21.15.105:0/1346521780 >> 172.21.15.105:0/1346521780 conn(0x7f42ec0fc400 msgr2=0x7f42ec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- 172.21.15.105:0/1346521780 shutdown_connections 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- 172.21.15.105:0/1346521780 wait complete. 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 Processor -- start 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- start start 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10db60 con 0x7f42ec102a90 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10d9e0 con 0x7f42ec1053c0 2024-04-22T06:37:28.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f4775640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec10dce0 con 0x7f42ec107cf0 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56250/0 (socket says 172.21.15.105:56250) 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.209+0000 7f42f2511640 1 -- 172.21.15.105:0/831452779 learned_addr learned my addr 172.21.15.105:0/831452779 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2d12640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f1d10640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 -- 172.21.15.105:0/831452779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 msgr2=0x7f42ec100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 -- 172.21.15.105:0/831452779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 msgr2=0x7f42ec100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 -- 172.21.15.105:0/831452779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec101840 con 0x7f42ec102a90 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f1d10640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2d12640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:28.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f2511640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f42dc006010 tx=0x7f42dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42e37fe640 1 -- 172.21.15.105:0/831452779 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc043020 con 0x7f42ec102a90 2024-04-22T06:37:28.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42e37fe640 1 -- 172.21.15.105:0/831452779 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42dc004aa0 con 0x7f42ec102a90 2024-04-22T06:37:28.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ec1a23c0 con 0x7f42ec102a90 2024-04-22T06:37:28.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42e37fe640 1 -- 172.21.15.105:0/831452779 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc003c70 con 0x7f42ec102a90 2024-04-22T06:37:28.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.210+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ec1a2960 con 0x7f42ec102a90 2024-04-22T06:37:28.212 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:27 smithi112 ceph-mon[31236]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:28.213 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:27 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1246998362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.213 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:27 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2744984899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.213 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:27 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2863982133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.212+0000 7f42e37fe640 1 -- 172.21.15.105:0/831452779 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f42dc046020 con 0x7f42ec102a90 2024-04-22T06:37:28.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.212+0000 7f42f7366740 1 Processor -- start 2024-04-22T06:37:28.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 -- start start 2024-04-22T06:37:28.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42e37fe640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e26689fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e266803b10 con 0x55e26689ee40 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2667a1240 con 0x55e26689f830 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f7366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e2667ffad0 con 0x55e2668a02d0 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2d12640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42e37fe640 1 -- 172.21.15.105:0/831452779 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f42dc0f27e0 con 0x7f42ec102a90 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56256/0 (socket says 172.21.15.105:56256) 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 -- 172.21.15.105:0/3859636801 learned_addr learned my addr 172.21.15.105:0/3859636801 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2d12640 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f1d10640 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e26689fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 -- 172.21.15.105:0/3859636801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 msgr2=0x55e2668a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668a6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 -- 172.21.15.105:0/3859636801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 msgr2=0x55e26689fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e26689fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 -- 172.21.15.105:0/3859636801 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2665162d0 con 0x55e26689ee40 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2d12640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 0x7f42d80a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f42e4005fd0 tx=0x7f42e4009e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.213+0000 7f42f2511640 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0f0380 tx=0x7f42dc0be4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7191d842764edc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42e0ff9640 1 -- 172.21.15.105:0/3859636801 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0044e0 con 0x55e26689ee40 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42e0ff9640 1 -- 172.21.15.105:0/3859636801 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42dc0035f0 con 0x55e26689ee40 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42e0ff9640 1 -- 172.21.15.105:0/3859636801 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0c77d0 con 0x55e26689ee40 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 -- 172.21.15.105:0/3859636801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 msgr2=0x55e26689f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0f0380 tx=0x7f42dc0be4c0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 -- 172.21.15.105:0/3859636801 shutdown_connections 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e26689fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 --2- 172.21.15.105:0/3859636801 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e26689f260 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 -- 172.21.15.105:0/3859636801 >> 172.21.15.105:0/3859636801 conn(0x55e26689af40 msgr2=0x55e266899490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 -- 172.21.15.105:0/3859636801 shutdown_connections 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.214+0000 7f42f7366740 1 -- 172.21.15.105:0/3859636801 wait complete. 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 -- start start 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 Processor -- start 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689ee40 0x55e2668a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668b0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e2668a0ad0 con 0x55e26689f830 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e26678c190 con 0x55e26689ee40 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e2668a73f0 con 0x55e2668a02d0 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f7366740 10 client.0 _ll_register_callbacks cb 0x55e2667878b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56272/0 (socket says 172.21.15.105:56272) 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 -- 172.21.15.105:0/1043091195 learned_addr learned my addr 172.21.15.105:0/1043091195 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f2511640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689ee40 0x55e2668a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f2d12640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 -- 172.21.15.105:0/1043091195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 msgr2=0x55e2668b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 -- 172.21.15.105:0/1043091195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689ee40 msgr2=0x55e2668a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689ee40 0x55e2668a8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f1d10640 1 -- 172.21.15.105:0/1043091195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2665e06b0 con 0x55e26689f830 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.215+0000 7f42f2d12640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e2d0 tx=0x7f42e800e790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cf684ddc6e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8018030 con 0x55e26689f830 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f42e800f040 con 0x55e26689f830 2024-04-22T06:37:28.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e266811530 con 0x55e26689f830 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f42e8013660 con 0x55e26689f830 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 10 client.15255 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e2668b35a0 con 0x55e26689f830 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e266811530 con 0x55e26689f830 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f42e801d930 con 0x55e26689f830 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 10 client.15255 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 10 client.15255 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 20 client.15255 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e2668b5ce0 0x55e2668b7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 4 client.15255 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e26680b3a0 con 0x55e2668b5ce0 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.216+0000 7f42f2511640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e2668b5ce0 0x55e2668b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f2511640 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e2668b5ce0 0x55e2668b7150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e12bba3c76f44272 server_cookie=76f59bdf91c42f5f in_seq=0 out_seq=0 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42d5ffb640 10 client.15255 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:28.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1043091195 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e26680b3a0 con 0x55e2668b5ce0 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42d5ffb640 10 client.15255 handle_command_reply: tid=0 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 client.15255 shutdown 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 msgr2=0x55e2668a9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f42e800e2d0 tx=0x7f42e800e790 comp rx=0 tx=0).stop 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 shutdown_connections 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e2668b5ce0 0x55e2668b7150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e2668a02d0 0x55e2668b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689f830 0x55e2668a9120 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 --2- 172.21.15.105:0/1043091195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689ee40 0x55e2668a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42d5ffb640 0 client.15255 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.217+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 >> 172.21.15.105:0/1043091195 conn(0x55e26689af40 msgr2=0x55e266898a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.218+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 shutdown_connections 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.218+0000 7f42f7366740 1 -- 172.21.15.105:0/1043091195 wait complete. 2024-04-22T06:37:28.225 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.218+0000 7f42f7366740 20 client.15255 trim_cache size 0 max 16384 2024-04-22T06:37:28.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 Processor -- start 2024-04-22T06:37:28.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 -- start start 2024-04-22T06:37:28.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689ee40 0x55e2668a3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e2668a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b5490 con 0x55e2668a02d0 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.235+0000 7f42f7366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b5b60 con 0x55e26689f830 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f7366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b6230 con 0x55e26689ee40 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56274/0 (socket says 172.21.15.105:56274) 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 -- 172.21.15.105:0/4221022116 learned_addr learned my addr 172.21.15.105:0/4221022116 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2511640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689ee40 0x55e2668a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f1d10640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e2668a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 -- 172.21.15.105:0/4221022116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689ee40 msgr2=0x55e2668a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689ee40 0x55e2668a3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 -- 172.21.15.105:0/4221022116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 msgr2=0x55e2668a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e2668a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 -- 172.21.15.105:0/4221022116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e26689a230 con 0x55e2668a02d0 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f1d10640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e2668a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42f2d12640 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009980 tx=0x7f42e40094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb0956a678171d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42d7fff640 1 -- 172.21.15.105:0/4221022116 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4026070 con 0x55e2668a02d0 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42d7fff640 1 -- 172.21.15.105:0/4221022116 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e4009b90 con 0x55e2668a02d0 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.236+0000 7f42d7fff640 1 -- 172.21.15.105:0/4221022116 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4022c20 con 0x55e2668a02d0 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- 172.21.15.105:0/4221022116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 msgr2=0x55e2668b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009980 tx=0x7f42e40094a0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- 172.21.15.105:0/4221022116 shutdown_connections 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e2668a02d0 0x55e2668b0120 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e26689f830 0x55e2668a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- 172.21.15.105:0/4221022116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689ee40 0x55e2668a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- 172.21.15.105:0/4221022116 >> 172.21.15.105:0/4221022116 conn(0x55e26689af40 msgr2=0x55e266899440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- 172.21.15.105:0/4221022116 shutdown_connections 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- 172.21.15.105:0/4221022116 wait complete. 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- start start 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 Processor -- start 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689f830 0x55e2668ae110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2668a02d0 0x55e266919c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b6fd0 con 0x55e26689ee40 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b6900 con 0x55e2668a02d0 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e2668b76a0 con 0x55e26689f830 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f7366740 10 client.0 _ll_register_callbacks cb 0x7f42e800c190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f2511640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56288/0 (socket says 172.21.15.105:56288) 2024-04-22T06:37:28.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f2511640 1 -- 172.21.15.105:0/1901117997 learned_addr learned my addr 172.21.15.105:0/1901117997 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.237+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689f830 0x55e2668ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 -- 172.21.15.105:0/1901117997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689f830 msgr2=0x55e2668ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689f830 0x55e2668ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 -- 172.21.15.105:0/1901117997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2668a02d0 msgr2=0x55e266919c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2d12640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2668a02d0 0x55e266919c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2668a02d0 0x55e266919c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 -- 172.21.15.105:0/1901117997 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e26691a430 con 0x55e26689ee40 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f2511640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0bafc0 tx=0x7f42dc0df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9be9d0ebb9f8ad server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0beb60 con 0x55e26689ee40 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f42dc0bcc00 con 0x55e26689ee40 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e266787960 con 0x55e26689ee40 2024-04-22T06:37:28.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f42dc0bf260 con 0x55e26689ee40 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 10 client.15261 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e26691b6a0 con 0x55e26689ee40 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e266787960 con 0x55e26689ee40 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f42dc0c7340 con 0x55e26689ee40 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 10 client.15261 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 10 client.15261 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 20 client.15261 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e266920100 0x55e26691ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 4 client.15261 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.238+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "status"}) -- 0x55e2667d4520 con 0x55e266920100 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e266920100 0x55e26691ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42d5ffb640 10 client.15261 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f1d10640 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e266920100 0x55e26691ee10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0c83536ad1d9b7d server_cookie=a35de33fc46fe39e in_seq=0 out_seq=0 2024-04-22T06:37:28.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42d5ffb640 1 -- 172.21.15.105:0/1901117997 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55e2667d4520 con 0x55e266920100 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42d5ffb640 10 client.15261 handle_command_reply: tid=0 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 client.15261 shutdown 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 msgr2=0x55e2668a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0bafc0 tx=0x7f42dc0df040 comp rx=0 tx=0).stop 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 shutdown_connections 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e2668a02d0 0x55e266919c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55e266920100 0x55e26691ee10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e26689f830 0x55e2668ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 --2- 172.21.15.105:0/1901117997 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e26689ee40 0x55e2668a2e40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42d5ffb640 0 client.15261 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 >> 172.21.15.105:0/1901117997 conn(0x55e26689af40 msgr2=0x55e2668a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 shutdown_connections 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 1 -- 172.21.15.105:0/1901117997 wait complete. 2024-04-22T06:37:28.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.239+0000 7f42f7366740 20 client.15261 trim_cache size 0 max 16384 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 4, 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 14640, 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 102.240421883, 2024-04-22T06:37:28.246 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:28.247 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:28.247 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:28.247 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 115.830388391 2024-04-22T06:37:28.247 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:28.247 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 msgr2=0x7f42d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 0x7f42d80a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f42e4005fd0 tx=0x7f42e4009e60 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 msgr2=0x7f42ec1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f42dc006010 tx=0x7f42dc0042f0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 shutdown_connections 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f42d80a3bb0 0x7f42d80a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f42ec107cf0 0x7f42ec100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f42ec1053c0 0x7f42ec100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 --2- 172.21.15.105:0/831452779 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42ec102a90 0x7f42ec1023b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 >> 172.21.15.105:0/831452779 conn(0x7f42ec0fc400 msgr2=0x7f42ec103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 shutdown_connections 2024-04-22T06:37:28.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.240+0000 7f42f4775640 1 -- 172.21.15.105:0/831452779 wait complete. 2024-04-22T06:37:28.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:27 smithi177 ceph-mon[31346]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-22T06:37:28.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:27 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1246998362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:27 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2744984899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.249 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:27 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2863982133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:28.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:37:28.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- 172.21.15.105:0/1399338610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102a00 msgr2=0x7fc1ec109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/1399338610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102a00 0x7fc1ec109320 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009ca0 tx=0x7fc1e402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- 172.21.15.105:0/1399338610 shutdown_connections 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/1399338610 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102a00 0x7fc1ec109320 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/1399338610 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec102040 0x7fc1ec1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/1399338610 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec100e60 0x7fc1ec101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- 172.21.15.105:0/1399338610 >> 172.21.15.105:0/1399338610 conn(0x7fc1ec0fc440 msgr2=0x7fc1ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- 172.21.15.105:0/1399338610 shutdown_connections 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- 172.21.15.105:0/1399338610 wait complete. 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 Processor -- start 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.350+0000 7fc1f2c11640 1 -- start start 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec100e60 0x7fc1ec197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 0x7fc1ec19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10ca00 con 0x7fc1ec102040 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10c880 con 0x7fc1ec102a00 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec10cb80 con 0x7fc1ec100e60 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56312/0 (socket says 172.21.15.105:56312) 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3140429228 learned_addr learned my addr 172.21.15.105:0/3140429228 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec100e60 0x7fc1ec197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 0x7fc1ec19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3140429228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec100e60 msgr2=0x7fc1ec197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec100e60 0x7fc1ec197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3140429228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 msgr2=0x7fc1ec19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 0x7fc1ec19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3140429228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ec19ec90 con 0x7fc1ec102040 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 0x7fc1ec19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80078f0 tx=0x7fc1d8007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1e9ffb640 1 -- 172.21.15.105:0/3140429228 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8019020 con 0x7fc1ec102040 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ec06a5f0 con 0x7fc1ec102040 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1e9ffb640 1 -- 172.21.15.105:0/3140429228 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1d80054d0 con 0x7fc1ec102040 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.351+0000 7fc1e9ffb640 1 -- 172.21.15.105:0/3140429228 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d800e650 con 0x7fc1ec102040 2024-04-22T06:37:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.352+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ec06aaf0 con 0x7fc1ec102040 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.354+0000 7fc1e9ffb640 1 -- 172.21.15.105:0/3140429228 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fc1d8005020 con 0x7fc1ec102040 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.354+0000 7fc1f5803740 1 Processor -- start 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.354+0000 7fc1f5803740 1 -- start start 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1e9ffb640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 0x7fc1dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584b021eb10 con 0x5584b02be250 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5584b01bc240 con 0x5584b02bf6e0 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f5803740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584b021aad0 con 0x5584b02bec40 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56320/0 (socket says 172.21.15.105:56320) 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 -- 172.21.15.105:0/3540631940 learned_addr learned my addr 172.21.15.105:0/3540631940 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 0x7fc1dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1e9ffb640 1 -- 172.21.15.105:0/3140429228 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc1d80d1a30 con 0x7fc1ec102040 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 -- 172.21.15.105:0/3540631940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 msgr2=0x5584b02bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02bf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 -- 172.21.15.105:0/3540631940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 msgr2=0x5584b02c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 -- 172.21.15.105:0/3540631940 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584aff312d0 con 0x5584b02be250 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 0x7fc1dc0a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4033000 tx=0x7fc1e4005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009430 tx=0x7fc1e4045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3b88a50023437d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.355+0000 7fc1d6ffd640 1 -- 172.21.15.105:0/3540631940 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4039070 con 0x5584b02be250 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1d6ffd640 1 -- 172.21.15.105:0/3540631940 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e403a040 con 0x5584b02be250 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1d6ffd640 1 -- 172.21.15.105:0/3540631940 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4034900 con 0x5584b02be250 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 -- 172.21.15.105:0/3540631940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 msgr2=0x5584b02be670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009430 tx=0x7fc1e4045000 comp rx=0 tx=0).stop 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 -- 172.21.15.105:0/3540631940 shutdown_connections 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3540631940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02be670 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 -- 172.21.15.105:0/3540631940 >> 172.21.15.105:0/3540631940 conn(0x5584b02b5f40 msgr2=0x5584b02b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 -- 172.21.15.105:0/3540631940 shutdown_connections 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.356+0000 7fc1f5803740 1 -- 172.21.15.105:0/3540631940 wait complete. 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 -- start start 2024-04-22T06:37:28.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 Processor -- start 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02c8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584b02c6860 con 0x5584b02be250 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5584b01a7190 con 0x5584b02bf6e0 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584b02c69e0 con 0x5584b02bec40 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f5803740 10 client.0 _ll_register_callbacks cb 0x5584b01a28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56330/0 (socket says 172.21.15.105:56330) 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 -- 172.21.15.105:0/3239453960 learned_addr learned my addr 172.21.15.105:0/3239453960 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 -- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 msgr2=0x5584b02c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 -- 172.21.15.105:0/3239453960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 msgr2=0x5584b02cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f09ad640 1 -- 172.21.15.105:0/3239453960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584afffb6b0 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.357+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009d70 tx=0x7fc1e00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36faa92ca907a9b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0005220 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc1e00053c0 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e403a040 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1e0011e60 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 10 client.15273 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5584b02d2c50 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e403a040 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc1e000f040 con 0x5584b02be250 2024-04-22T06:37:28.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 10 client.15273 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 10 client.15273 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 20 client.15273 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02d52f0 0x5584b02d6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 4 client.15273 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.358+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584b022c530 con 0x5584b02d52f0 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.359+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02d52f0 0x5584b02d6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.359+0000 7fc1cbfff640 10 client.15273 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.359+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02d52f0 0x5584b02d6760 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=21c1721e3f50fc98 server_cookie=40fd9ec4118f79eb in_seq=0 out_seq=0 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3239453960 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584b022c530 con 0x5584b02d52f0 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1cbfff640 10 client.15273 handle_command_reply: tid=0 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 client.15273 shutdown 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 msgr2=0x5584b02c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009d70 tx=0x7fc1e00117b0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 shutdown_connections 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02d52f0 0x5584b02d6760 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bf6e0 0x5584b02cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1cbfff640 0 client.15273 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3239453960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02c8240 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 >> 172.21.15.105:0/3239453960 conn(0x5584b02b5f40 msgr2=0x5584b02b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 shutdown_connections 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 1 -- 172.21.15.105:0/3239453960 wait complete. 2024-04-22T06:37:28.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.360+0000 7fc1f5803740 20 client.15273 trim_cache size 0 max 16384 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 Processor -- start 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 -- start start 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02be250 0x5584b02cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d4aa0 con 0x5584b02b5f40 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d5170 con 0x5584b02be250 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f5803740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d5840 con 0x5584b02bec40 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56346/0 (socket says 172.21.15.105:56346) 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 -- 172.21.15.105:0/2441546550 learned_addr learned my addr 172.21.15.105:0/2441546550 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 -- 172.21.15.105:0/2441546550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 msgr2=0x5584b02b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f11ae640 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 -- 172.21.15.105:0/2441546550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02be250 msgr2=0x5584b02cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02be250 0x5584b02cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02be250 0x5584b02cf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 -- 172.21.15.105:0/2441546550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b02c9370 con 0x5584b02b5f40 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0001eb0 tx=0x7fc1e0011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73031ea757985123 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1d67fc640 1 -- 172.21.15.105:0/2441546550 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0012660 con 0x5584b02b5f40 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1d67fc640 1 -- 172.21.15.105:0/2441546550 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e0012800 con 0x5584b02b5f40 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.378+0000 7fc1d67fc640 1 -- 172.21.15.105:0/2441546550 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0004c40 con 0x5584b02b5f40 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- 172.21.15.105:0/2441546550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 msgr2=0x5584b02c22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0001eb0 tx=0x7fc1e0011cb0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- 172.21.15.105:0/2441546550 shutdown_connections 2024-04-22T06:37:28.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02bec40 0x5584b02b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02be250 0x5584b02cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 --2- 172.21.15.105:0/2441546550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02b5f40 0x5584b02c22a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- 172.21.15.105:0/2441546550 >> 172.21.15.105:0/2441546550 conn(0x5584b02b4790 msgr2=0x5584b02c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- 172.21.15.105:0/2441546550 shutdown_connections 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- 172.21.15.105:0/2441546550 wait complete. 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 -- start start 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.379+0000 7fc1f5803740 1 Processor -- start 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 0x5584b02cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bec40 0x5584b02ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d65e0 con 0x5584b02be250 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d5f10 con 0x5584b02bec40 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5584b02d6cb0 con 0x5584b02b5f40 2024-04-22T06:37:28.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 10 client.0 _ll_register_callbacks cb 0x7fc1e0009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56358/0 (socket says 172.21.15.105:56358) 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3172341558 learned_addr learned my addr 172.21.15.105:0/3172341558 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 0x5584b02cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 msgr2=0x5584b02cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 0x5584b02cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3172341558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bec40 msgr2=0x5584b02ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bec40 0x5584b02ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 -- 172.21.15.105:0/3172341558 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b02cedd0 con 0x5584b02be250 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 0x5584b02cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1ebfff640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80cf5d0 tx=0x7fc1d800c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d160e8621cf0b996 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80a0070 con 0x5584b02be250 2024-04-22T06:37:28.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fc1d809bee0 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e0012800 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1d809e540 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 10 client.15279 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.380+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5584b033b630 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1e0012800 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fc1d8014070 con 0x5584b02be250 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 10 client.15279 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 10 client.15279 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 20 client.15279 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02bf6e0 0x5584b033efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 4 client.15279 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "status"}) -- 0x5584b02263a0 con 0x5584b02bf6e0 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02bf6e0 0x5584b033efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1cbfff640 10 client.15279 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.381+0000 7fc1f09ad640 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02bf6e0 0x5584b033efc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=18ac96de7317d2c7 server_cookie=2e4e1745587bfd63 in_seq=0 out_seq=0 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1cbfff640 1 -- 172.21.15.105:0/3172341558 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5584b02263a0 con 0x5584b02bf6e0 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1cbfff640 10 client.15279 handle_command_reply: tid=0 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 client.15279 shutdown 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 msgr2=0x5584b02cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80cf5d0 tx=0x7fc1d800c510 comp rx=0 tx=0).stop 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 shutdown_connections 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5584b02bf6e0 0x5584b033efc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5584b02bec40 0x5584b02ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5584b02be250 0x5584b02cb7c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1cbfff640 0 client.15279 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 --2- 172.21.15.105:0/3172341558 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5584b02b5f40 0x5584b02cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 >> 172.21.15.105:0/3172341558 conn(0x5584b02b4790 msgr2=0x5584b02b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 shutdown_connections 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 1 -- 172.21.15.105:0/3172341558 wait complete. 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f5803740 20 client.15279 trim_cache size 0 max 16384 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 3, 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24454, 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 103.346332233, 2024-04-22T06:37:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 126.86569040800001 2024-04-22T06:37:28.389 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 msgr2=0x7fc1dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 0x7fc1dc0a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4033000 tx=0x7fc1e4005ef0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 msgr2=0x7fc1ec1975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.382+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80078f0 tx=0x7fc1d8007db0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 shutdown_connections 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fc1dc0a3bb0 0x7fc1dc0a6070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ec102a00 0x7fc1ec19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1ec102040 0x7fc1ec1975b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 --2- 172.21.15.105:0/3140429228 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc1ec100e60 0x7fc1ec197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 >> 172.21.15.105:0/3140429228 conn(0x7fc1ec0fc440 msgr2=0x7fc1ec1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 shutdown_connections 2024-04-22T06:37:28.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.383+0000 7fc1f2c11640 1 -- 172.21.15.105:0/3140429228 wait complete. 2024-04-22T06:37:28.392 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:37:28.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 -- 172.21.15.105:0/1582082590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90100e60 msgr2=0x7fee90101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 --2- 172.21.15.105:0/1582082590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90100e60 0x7fee90101280 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fee80009900 tx=0x7fee8002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 -- 172.21.15.105:0/1582082590 shutdown_connections 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 --2- 172.21.15.105:0/1582082590 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 0x7fee90109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 --2- 172.21.15.105:0/1582082590 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90102040 0x7fee901024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 --2- 172.21.15.105:0/1582082590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90100e60 0x7fee90101280 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 -- 172.21.15.105:0/1582082590 >> 172.21.15.105:0/1582082590 conn(0x7fee900fc420 msgr2=0x7fee900fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 -- 172.21.15.105:0/1582082590 shutdown_connections 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.492+0000 7fee98532640 1 -- 172.21.15.105:0/1582082590 wait complete. 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 Processor -- start 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 -- start start 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90100e60 0x7fee90198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 0x7fee9019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010db30 con 0x7fee90102040 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010d9b0 con 0x7fee90100e60 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee98532640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fee9010dcb0 con 0x7fee90102a00 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56368/0 (socket says 172.21.15.105:56368) 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 -- 172.21.15.105:0/3960913570 learned_addr learned my addr 172.21.15.105:0/3960913570 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee962ce640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90100e60 0x7fee90198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee96acf640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 0x7fee9019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 -- 172.21.15.105:0/3960913570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 msgr2=0x7fee9019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 0x7fee9019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 -- 172.21.15.105:0/3960913570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90100e60 msgr2=0x7fee90198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90100e60 0x7fee90198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 -- 172.21.15.105:0/3960913570 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9019fe60 con 0x7fee90102040 2024-04-22T06:37:28.494 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee95acd640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00c890 tx=0x7fee8c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.493+0000 7fee877fe640 1 -- 172.21.15.105:0/3960913570 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c004220 con 0x7fee90102040 2024-04-22T06:37:28.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.494+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee901a0150 con 0x7fee90102040 2024-04-22T06:37:28.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.494+0000 7fee877fe640 1 -- 172.21.15.105:0/3960913570 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8c0043c0 con 0x7fee90102040 2024-04-22T06:37:28.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.494+0000 7fee877fe640 1 -- 172.21.15.105:0/3960913570 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c004a00 con 0x7fee90102040 2024-04-22T06:37:28.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.494+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee901a06f0 con 0x7fee90102040 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.496+0000 7fee877fe640 1 -- 172.21.15.105:0/3960913570 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fee8c0079b0 con 0x7fee90102040 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.496+0000 7fee9b1ff740 1 Processor -- start 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 -- start start 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a87260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a8eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a8959ebb10 con 0x55a895a87830 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a895989240 con 0x55a895a882d0 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a8959e7ad0 con 0x55a895a86e40 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee877fe640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 0x7fee7c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56380/0 (socket says 172.21.15.105:56380) 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 -- 172.21.15.105:0/446424917 learned_addr learned my addr 172.21.15.105:0/446424917 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee96acf640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 0x7fee7c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee95acd640 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a8eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee96acf640 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a87260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee877fe640 1 -- 172.21.15.105:0/3960913570 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fee8c0d0140 con 0x7fee90102040 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 -- 172.21.15.105:0/446424917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 msgr2=0x55a895a87260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a87260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 -- 172.21.15.105:0/446424917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 msgr2=0x55a895a8eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a8eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee962ce640 1 -- 172.21.15.105:0/446424917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8956fe2d0 con 0x55a895a87830 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.497+0000 7fee96acf640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 0x7fee7c0a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fee88009fd0 tx=0x7fee88009d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee962ce640 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fee80009900 tx=0x7fee80005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1460e56b799bf3e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee84bfe640 1 -- 172.21.15.105:0/446424917 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80045020 con 0x55a895a87830 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee84bfe640 1 -- 172.21.15.105:0/446424917 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8002caa0 con 0x55a895a87830 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee84bfe640 1 -- 172.21.15.105:0/446424917 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee800046c0 con 0x55a895a87830 2024-04-22T06:37:28.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 -- 172.21.15.105:0/446424917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 msgr2=0x55a895a87cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fee80009900 tx=0x7fee80005e00 comp rx=0 tx=0).stop 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 -- 172.21.15.105:0/446424917 shutdown_connections 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a8eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a87cb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/446424917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a87260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.498+0000 7fee9b1ff740 1 -- 172.21.15.105:0/446424917 >> 172.21.15.105:0/446424917 conn(0x55a895a82f40 msgr2=0x55a895a81490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- 172.21.15.105:0/446424917 shutdown_connections 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- 172.21.15.105:0/446424917 wait complete. 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- start start 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 Processor -- start 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a90da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a98320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a895a8f450 con 0x55a895a87830 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a895974190 con 0x55a895a882d0 2024-04-22T06:37:28.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a895a8f5d0 con 0x55a895a86e40 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee9b1ff740 10 client.0 _ll_register_callbacks cb 0x55a89596f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56394/0 (socket says 172.21.15.105:56394) 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 -- 172.21.15.105:0/3055454299 learned_addr learned my addr 172.21.15.105:0/3055454299 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee96acf640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a98320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee962ce640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a90da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 -- 172.21.15.105:0/3055454299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 msgr2=0x55a895a90da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a90da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 -- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 msgr2=0x55a895a98320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a98320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee95acd640 1 -- 172.21.15.105:0/3055454299 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8957c86b0 con 0x55a895a87830 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.499+0000 7fee96acf640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a98320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee95acd640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee8c098df0 tx=0x7fee8c01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958a604109437ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8c005290 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fee8c01ed20 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee8002caa0 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee8c09cb20 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 10 client.15291 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a895a9b700 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8002caa0 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fee8c01cc40 con 0x55a895a87830 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 10 client.15291 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 20 client.15291 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895a9dd60 0x55a895a9f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 4 client.15291 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.500+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8959f9530 con 0x55a895a9dd60 2024-04-22T06:37:28.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.501+0000 7fee962ce640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895a9dd60 0x55a895a9f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.501+0000 7fee79ffb640 10 client.15291 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.501+0000 7fee962ce640 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895a9dd60 0x55a895a9f1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b7a0777431a6802 server_cookie=5f0136359518b5a9 in_seq=0 out_seq=0 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee79ffb640 1 -- 172.21.15.105:0/3055454299 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8959f9530 con 0x55a895a9dd60 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee79ffb640 10 client.15291 handle_command_reply: tid=0 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 client.15291 shutdown 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 msgr2=0x55a895a912e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee8c098df0 tx=0x7fee8c01ecb0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 shutdown_connections 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a882d0 0x55a895a98320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a87830 0x55a895a912e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895a9dd60 0x55a895a9f1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3055454299 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a90da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee79ffb640 0 client.15291 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 >> 172.21.15.105:0/3055454299 conn(0x55a895a82f40 msgr2=0x55a895a80a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 shutdown_connections 2024-04-22T06:37:28.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3055454299 wait complete. 2024-04-22T06:37:28.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.502+0000 7fee9b1ff740 20 client.15291 trim_cache size 0 max 16384 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 Processor -- start 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 -- start start 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a8ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9d510 con 0x55a895a882d0 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9dbe0 con 0x55a895a87830 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9e2b0 con 0x55a895a86e40 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56396/0 (socket says 172.21.15.105:56396) 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 -- 172.21.15.105:0/3728967591 learned_addr learned my addr 172.21.15.105:0/3728967591 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee962ce640 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a8ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 -- 172.21.15.105:0/3728967591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 msgr2=0x55a895a8ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a8ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 -- 172.21.15.105:0/3728967591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 msgr2=0x55a895a982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 -- 172.21.15.105:0/3728967591 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a895a8ea20 con 0x55a895a882d0 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee96acf640 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee88012600 tx=0x7fee88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2eaa6c1ab2cf354 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee7bfff640 1 -- 172.21.15.105:0/3728967591 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8801fa50 con 0x55a895a882d0 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee7bfff640 1 -- 172.21.15.105:0/3728967591 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8801fbf0 con 0x55a895a882d0 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.520+0000 7fee7bfff640 1 -- 172.21.15.105:0/3728967591 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee880283a0 con 0x55a895a882d0 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3728967591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 msgr2=0x55a895a90da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fee88012600 tx=0x7fee88004a40 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3728967591 shutdown_connections 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a882d0 0x55a895a90da0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/3728967591 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a86e40 0x55a895a8ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3728967591 >> 172.21.15.105:0/3728967591 conn(0x55a895a82f40 msgr2=0x55a895a7c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3728967591 shutdown_connections 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- 172.21.15.105:0/3728967591 wait complete. 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 -- start start 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 Processor -- start 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.521+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a93b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a882d0 0x55a895a96b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9f050 con 0x55a895a86e40 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9e980 con 0x55a895a87830 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a895a9f720 con 0x55a895a882d0 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 10 client.0 _ll_register_callbacks cb 0x7fee8c005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56412/0 (socket says 172.21.15.105:56412) 2024-04-22T06:37:28.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 -- 172.21.15.105:0/1638862090 learned_addr learned my addr 172.21.15.105:0/1638862090 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee96acf640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a882d0 0x55a895a96b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee95acd640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a93b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 -- 172.21.15.105:0/1638862090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a882d0 msgr2=0x55a895a96b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a882d0 0x55a895a96b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 -- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 msgr2=0x55a895a93b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a93b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 -- 172.21.15.105:0/1638862090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a895a97110 con 0x55a895a86e40 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee95acd640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a93b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee962ce640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fee80006010 tx=0x7fee80005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=531d527287abc873 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80045040 con 0x55a895a86e40 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fee80004240 con 0x55a895a86e40 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee8801fbf0 con 0x55a895a86e40 2024-04-22T06:37:28.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee80004440 con 0x55a895a86e40 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 10 client.15297 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a895a98410 con 0x55a895a86e40 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.522+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee8801fbf0 con 0x55a895a86e40 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fee800038f0 con 0x55a895a86e40 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 10 client.15297 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 10 client.15297 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 20 client.15297 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895b080c0 0x55a895b06dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 4 client.15297 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "status"}) -- 0x55a8959f33a0 con 0x55a895b080c0 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee95acd640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895b080c0 0x55a895b06dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee79ffb640 10 client.15297 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.523+0000 7fee95acd640 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895b080c0 0x55a895b06dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89ddf2c5ed65d832 server_cookie=6245d225a2b99c63 in_seq=0 out_seq=0 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee79ffb640 1 -- 172.21.15.105:0/1638862090 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a8959f33a0 con 0x55a895b080c0 2024-04-22T06:37:28.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee79ffb640 10 client.15297 handle_command_reply: tid=0 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 client.15297 shutdown 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 msgr2=0x55a895a935d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fee80006010 tx=0x7fee80005b70 comp rx=0 tx=0).stop 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 shutdown_connections 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x55a895b080c0 0x55a895b06dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a895a882d0 0x55a895a96b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a895a87830 0x55a895a93b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee79ffb640 0 client.15297 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 --2- 172.21.15.105:0/1638862090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a895a86e40 0x55a895a935d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 >> 172.21.15.105:0/1638862090 conn(0x55a895a82f40 msgr2=0x55a895a90da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 shutdown_connections 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 1 -- 172.21.15.105:0/1638862090 wait complete. 2024-04-22T06:37:28.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee9b1ff740 20 client.15297 trim_cache size 0 max 16384 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 2, 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24461, 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 104.49939277199999, 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:28.530 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:28.531 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 124.276860083 2024-04-22T06:37:28.531 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:28.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 msgr2=0x7fee7c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 0x7fee7c0a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fee88009fd0 tx=0x7fee88009d00 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 msgr2=0x7fee90198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00c890 tx=0x7fee8c00cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 shutdown_connections 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fee7c0a3bb0 0x7fee7c0a6070 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fee90102a00 0x7fee9019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fee90102040 0x7fee90198780 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 --2- 172.21.15.105:0/3960913570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fee90100e60 0x7fee90198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 >> 172.21.15.105:0/3960913570 conn(0x7fee900fc420 msgr2=0x7fee90105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 shutdown_connections 2024-04-22T06:37:28.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.524+0000 7fee98532640 1 -- 172.21.15.105:0/3960913570 wait complete. 2024-04-22T06:37:28.534 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:37:28.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.633+0000 7f5f67481640 1 -- 172.21.15.105:0/387041555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60101fe0 msgr2=0x7f5f60102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.633+0000 7f5f67481640 1 --2- 172.21.15.105:0/387041555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60101fe0 0x7f5f60102460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009900 tx=0x7f5f5c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- 172.21.15.105:0/387041555 shutdown_connections 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- 172.21.15.105:0/387041555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f601029a0 0x7f5f601092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- 172.21.15.105:0/387041555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60101fe0 0x7f5f60102460 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- 172.21.15.105:0/387041555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f60100e90 0x7f5f601012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- 172.21.15.105:0/387041555 >> 172.21.15.105:0/387041555 conn(0x7f5f600fc400 msgr2=0x7f5f600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- 172.21.15.105:0/387041555 shutdown_connections 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- 172.21.15.105:0/387041555 wait complete. 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 Processor -- start 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- start start 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f60101fe0 0x7f5f60109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f601029a0 0x7f5f6010a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6010db30 con 0x7f5f60100e90 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6010d9b0 con 0x7f5f601029a0 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f67481640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6010dcb0 con 0x7f5f60101fe0 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f6521d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f6521d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56432/0 (socket says 172.21.15.105:56432) 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.634+0000 7f5f6521d640 1 -- 172.21.15.105:0/2633147850 learned_addr learned my addr 172.21.15.105:0/2633147850 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f60101fe0 0x7f5f60109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f601029a0 0x7f5f6010a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 -- 172.21.15.105:0/2633147850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f60101fe0 msgr2=0x7f5f60109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f60101fe0 0x7f5f60109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 -- 172.21.15.105:0/2633147850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f601029a0 msgr2=0x7f5f6010a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f601029a0 0x7f5f6010a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 -- 172.21.15.105:0/2633147850 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6010ac40 con 0x7f5f60100e90 2024-04-22T06:37:28.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d950 tx=0x7f5f5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f567fc640 1 -- 172.21.15.105:0/2633147850 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50017030 con 0x7f5f60100e90 2024-04-22T06:37:28.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f567fc640 1 -- 172.21.15.105:0/2633147850 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f50004020 con 0x7f5f60100e90 2024-04-22T06:37:28.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f601a2360 con 0x7f5f60100e90 2024-04-22T06:37:28.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f567fc640 1 -- 172.21.15.105:0/2633147850 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f500047a0 con 0x7f5f60100e90 2024-04-22T06:37:28.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.635+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f601a28d0 con 0x7f5f60100e90 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.637+0000 7f5f567fc640 1 -- 172.21.15.105:0/2633147850 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f5f50009a30 con 0x7f5f60100e90 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.637+0000 7f5f6a079740 1 Processor -- start 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.637+0000 7f5f6a079740 1 -- start start 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc0fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc16eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586dfb73b10 con 0x5586dfc0ee40 2024-04-22T06:37:28.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586dfb11240 con 0x5586dfc102d0 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6a079740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586dfb6fad0 con 0x5586dfc0f830 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56434/0 (socket says 172.21.15.105:56434) 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3050799430 learned_addr learned my addr 172.21.15.105:0/3050799430 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f6521d640 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc0fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc16eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f567fc640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 0x7f5f4c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3050799430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 msgr2=0x5586dfc0fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc0fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3050799430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 msgr2=0x5586dfc16eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc16eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3050799430 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586df8862d0 con 0x5586dfc0ee40 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 0x7f5f4c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f567fc640 1 -- 172.21.15.105:0/2633147850 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5f500cfed0 con 0x7f5f60100e90 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5f58010520 tx=0x7f5f580109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9cb2c24ac6a8b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f4b7fe640 1 -- 172.21.15.105:0/3050799430 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58019070 con 0x5586dfc0ee40 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.638+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 0x7f5f4c0a60f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5f58021000 tx=0x7f5f5801e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f4b7fe640 1 -- 172.21.15.105:0/3050799430 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f58003e10 con 0x5586dfc0ee40 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f4b7fe640 1 -- 172.21.15.105:0/3050799430 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5801d630 con 0x5586dfc0ee40 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- 172.21.15.105:0/3050799430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 msgr2=0x5586dfc0f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5f58010520 tx=0x7f5f580109e0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- 172.21.15.105:0/3050799430 shutdown_connections 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc16eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc0fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3050799430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc0f260 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- 172.21.15.105:0/3050799430 >> 172.21.15.105:0/3050799430 conn(0x5586dfc0af40 msgr2=0x5586dfc09490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- 172.21.15.105:0/3050799430 shutdown_connections 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- 172.21.15.105:0/3050799430 wait complete. 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.639+0000 7f5f6a079740 1 -- start start 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 Processor -- start 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc19460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc20430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc174b0 con 0x5586dfc0ee40 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586dfafc190 con 0x5586dfc102d0 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc17630 con 0x5586dfc0f830 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6a079740 10 client.0 _ll_register_callbacks cb 0x5586dfaf78b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56436/0 (socket says 172.21.15.105:56436) 2024-04-22T06:37:28.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 -- 172.21.15.105:0/1506503800 learned_addr learned my addr 172.21.15.105:0/1506503800 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc20430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc19460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 -- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 msgr2=0x5586dfc19460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc19460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 -- 172.21.15.105:0/1506503800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 msgr2=0x5586dfc20430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc20430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 -- 172.21.15.105:0/1506503800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586df9506b0 con 0x5586dfc0ee40 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc19460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f6521d640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5f50098c20 tx=0x7f5f5009c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7166a61e525551 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.640+0000 7f5f48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f500056a0 con 0x5586dfc0ee40 2024-04-22T06:37:28.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5f500041d0 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f58003e10 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5f5009c950 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 10 client.15309 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5586dfc238c0 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f58003e10 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5f500988c0 con 0x5586dfc0ee40 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 10 client.15309 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 20 client.15309 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc25e70 0x5586dfc272e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 4 client.15309 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.641+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586dfb81530 con 0x5586dfc25e70 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.642+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc25e70 0x5586dfc272e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.642+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc25e70 0x5586dfc272e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd2087d350eb4cd3 server_cookie=cdc96a9c8dca4a50 in_seq=0 out_seq=0 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.642+0000 7f5f48ff9640 10 client.15309 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f48ff9640 1 -- 172.21.15.105:0/1506503800 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586dfb81530 con 0x5586dfc25e70 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f48ff9640 10 client.15309 handle_command_reply: tid=0 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 client.15309 shutdown 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 msgr2=0x5586dfc18f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5f50098c20 tx=0x7f5f5009c0e0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 shutdown_connections 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc25e70 0x5586dfc272e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc102d0 0x5586dfc20430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f48ff9640 0 client.15309 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0f830 0x5586dfc19460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 --2- 172.21.15.105:0/1506503800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc0ee40 0x5586dfc18f20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 >> 172.21.15.105:0/1506503800 conn(0x5586dfc0af40 msgr2=0x5586dfc08a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 shutdown_connections 2024-04-22T06:37:28.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 1 -- 172.21.15.105:0/1506503800 wait complete. 2024-04-22T06:37:28.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.643+0000 7f5f6a079740 20 client.15309 trim_cache size 0 max 16384 2024-04-22T06:37:28.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.661+0000 7f5f6a079740 1 Processor -- start 2024-04-22T06:37:28.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.661+0000 7f5f6a079740 1 -- start start 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.661+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc09350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc09890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6a079740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc25620 con 0x5586dfc102d0 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6a079740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc25cf0 con 0x5586dfc0f830 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6a079740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc263c0 con 0x5586dfc0ee40 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56440/0 (socket says 172.21.15.105:56440) 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3529217557 learned_addr learned my addr 172.21.15.105:0/3529217557 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6521d640 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc09350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3529217557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 msgr2=0x5586dfc09350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc09350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3529217557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 msgr2=0x5586dfc09890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc09890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 -- 172.21.15.105:0/3529217557 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586dfc0ad40 con 0x5586dfc102d0 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f6521d640 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc09350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009bc0 tx=0x7f5f58009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d6cc5192a61e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f4affd640 1 -- 172.21.15.105:0/3529217557 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58035070 con 0x5586dfc102d0 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f4affd640 1 -- 172.21.15.105:0/3529217557 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f58003e10 con 0x5586dfc102d0 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.662+0000 7f5f4affd640 1 -- 172.21.15.105:0/3529217557 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58019050 con 0x5586dfc102d0 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- 172.21.15.105:0/3529217557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 msgr2=0x5586dfc203f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009bc0 tx=0x7f5f58009b50 comp rx=0 tx=0).stop 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- 172.21.15.105:0/3529217557 shutdown_connections 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc203f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc09890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 --2- 172.21.15.105:0/3529217557 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc09350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- 172.21.15.105:0/3529217557 >> 172.21.15.105:0/3529217557 conn(0x5586dfc0af40 msgr2=0x5586dfc13900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- 172.21.15.105:0/3529217557 shutdown_connections 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- 172.21.15.105:0/3529217557 wait complete. 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 -- start start 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 Processor -- start 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.663+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc1c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc1eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc27160 con 0x5586dfc102d0 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc26a90 con 0x5586dfc0f830 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586dfc27830 con 0x5586dfc0ee40 2024-04-22T06:37:28.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 10 client.0 _ll_register_callbacks cb 0x7f5f500d36e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56454/0 (socket says 172.21.15.105:56454) 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 -- 172.21.15.105:0/2547107116 learned_addr learned my addr 172.21.15.105:0/2547107116 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc1c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc1eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 -- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 msgr2=0x5586dfc1c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc1c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 -- 172.21.15.105:0/2547107116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 msgr2=0x5586dfc1eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc1eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 -- 172.21.15.105:0/2547107116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586dfc1f920 con 0x5586dfc102d0 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc1c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f64a1c640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc1eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f65a1e640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5f58003a90 tx=0x7f5f580132d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57af1adef35f4200 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58035040 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f5f58023040 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f58003e10 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5f58019020 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 10 client.15315 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.664+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5586dfc8bfa0 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f58003e10 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5f58038030 con 0x5586dfc102d0 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 10 client.15315 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 10 client.15315 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:28.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 20 client.15315 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc90400 0x5586dfc8f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 4 client.15315 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "status"}) -- 0x5586dfb7b3a0 con 0x5586dfc90400 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc90400 0x5586dfc8f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f6521d640 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc90400 0x5586dfc8f110 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73d943e02a8afdd3 server_cookie=e850c0b279f6cb88 in_seq=0 out_seq=0 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.665+0000 7f5f48ff9640 10 client.15315 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f48ff9640 1 -- 172.21.15.105:0/2547107116 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5586dfb7b3a0 con 0x5586dfc90400 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f48ff9640 10 client.15315 handle_command_reply: tid=0 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 client.15315 shutdown 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 msgr2=0x5586dfc1f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5f58003a90 tx=0x7f5f580132d0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 shutdown_connections 2024-04-22T06:37:28.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x5586dfc90400 0x5586dfc8f110 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586dfc102d0 0x5586dfc1f0f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586dfc0f830 0x5586dfc1eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f48ff9640 0 client.15315 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 --2- 172.21.15.105:0/2547107116 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586dfc0ee40 0x5586dfc1c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 >> 172.21.15.105:0/2547107116 conn(0x5586dfc0af40 msgr2=0x5586dfc16b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 shutdown_connections 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 1 -- 172.21.15.105:0/2547107116 wait complete. 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f6a079740 20 client.15315 trim_cache size 0 max 16384 2024-04-22T06:37:28.672 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 1, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24472, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 105.649401106, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 121.80438963 2024-04-22T06:37:28.673 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:28.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 msgr2=0x7f5f4c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 0x7f5f4c0a60f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5f58021000 tx=0x7f5f5801e000 comp rx=0 tx=0).stop 2024-04-22T06:37:28.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 msgr2=0x7f5f6010b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.666+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000d950 tx=0x7f5f5000de10 comp rx=0 tx=0).stop 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 shutdown_connections 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f5f4c0a3c30 0x7f5f4c0a60f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f601029a0 0x7f5f6010a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f60101fe0 0x7f5f60109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 --2- 172.21.15.105:0/2633147850 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5f60100e90 0x7f5f6010b830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 >> 172.21.15.105:0/2633147850 conn(0x7f5f600fc400 msgr2=0x7f5f60105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 shutdown_connections 2024-04-22T06:37:28.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.667+0000 7f5f67481640 1 -- 172.21.15.105:0/2633147850 wait complete. 2024-04-22T06:37:28.676 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:37:28.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 -- 172.21.15.105:0/233395913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 msgr2=0x7f4338102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 --2- 172.21.15.105:0/233395913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338102460 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f432c009900 tx=0x7f432c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 -- 172.21.15.105:0/233395913 shutdown_connections 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 --2- 172.21.15.105:0/233395913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43381029a0 0x7f43381092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 --2- 172.21.15.105:0/233395913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338102460 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 --2- 172.21.15.105:0/233395913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4338100e90 0x7f43381012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 -- 172.21.15.105:0/233395913 >> 172.21.15.105:0/233395913 conn(0x7f43380fc400 msgr2=0x7f43380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 -- 172.21.15.105:0/233395913 shutdown_connections 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.777+0000 7f433d904640 1 -- 172.21.15.105:0/233395913 wait complete. 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 Processor -- start 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 -- start start 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 0x7f4338198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43381029a0 0x7f433819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f433810c9f0 con 0x7f4338101fe0 2024-04-22T06:37:28.779 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f433810c870 con 0x7f4338100e90 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f433d904640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f433810cb70 con 0x7f43381029a0 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.778+0000 7f43367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56472/0 (socket says 172.21.15.105:56472) 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 -- 172.21.15.105:0/3141541876 learned_addr learned my addr 172.21.15.105:0/3141541876 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f4336ffd640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 0x7f4338198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43377fe640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43381029a0 0x7f433819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 -- 172.21.15.105:0/3141541876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43381029a0 msgr2=0x7f433819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43381029a0 0x7f433819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 -- 172.21.15.105:0/3141541876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 msgr2=0x7f4338198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 0x7f4338198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 -- 172.21.15.105:0/3141541876 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433819ff20 con 0x7f4338101fe0 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f4336ffd640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 0x7f4338198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:28.780 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f43367fc640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f432c036040 tx=0x7f432c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f433c902640 1 -- 172.21.15.105:0/3141541876 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c043020 con 0x7f4338101fe0 2024-04-22T06:37:28.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f433c902640 1 -- 172.21.15.105:0/3141541876 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f432c0052b0 con 0x7f4338101fe0 2024-04-22T06:37:28.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f433c902640 1 -- 172.21.15.105:0/3141541876 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c003cb0 con 0x7f4338101fe0 2024-04-22T06:37:28.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43381a01b0 con 0x7f4338101fe0 2024-04-22T06:37:28.781 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.779+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43381a06b0 con 0x7f4338101fe0 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.782+0000 7f43405d3740 1 Processor -- start 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.782+0000 7f43405d3740 1 -- start start 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad80260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad87eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56222ace4b10 con 0x56222ad80830 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56222ac82240 con 0x56222ad812d0 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43405d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56222ace0ad0 con 0x56222ad7fe40 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56486/0 (socket says 172.21.15.105:56486) 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 -- 172.21.15.105:0/735836776 learned_addr learned my addr 172.21.15.105:0/735836776 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43377fe640 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad80260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f43367fc640 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad87eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 -- 172.21.15.105:0/735836776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 msgr2=0x56222ad80260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad80260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 -- 172.21.15.105:0/735836776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 msgr2=0x56222ad87eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad87eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 -- 172.21.15.105:0/735836776 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222a9f72d0 con 0x56222ad80830 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f4336ffd640 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f43280103a0 tx=0x7f4328010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9747496a352f1f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.783+0000 7f431dffb640 1 -- 172.21.15.105:0/735836776 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328016070 con 0x56222ad80830 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f431dffb640 1 -- 172.21.15.105:0/735836776 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4328012bc0 con 0x56222ad80830 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f431dffb640 1 -- 172.21.15.105:0/735836776 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328011c10 con 0x56222ad80830 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 -- 172.21.15.105:0/735836776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 msgr2=0x56222ad80cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f43280103a0 tx=0x7f4328010860 comp rx=0 tx=0).stop 2024-04-22T06:37:28.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 -- 172.21.15.105:0/735836776 shutdown_connections 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad87eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad80cb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 --2- 172.21.15.105:0/735836776 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad80260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.784+0000 7f43405d3740 1 -- 172.21.15.105:0/735836776 >> 172.21.15.105:0/735836776 conn(0x56222ad7bf40 msgr2=0x56222ad7a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.785+0000 7f433c902640 1 -- 172.21.15.105:0/3141541876 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f432c046020 con 0x7f4338101fe0 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.785+0000 7f43405d3740 1 -- 172.21.15.105:0/735836776 shutdown_connections 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.785+0000 7f43405d3740 1 -- 172.21.15.105:0/735836776 wait complete. 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.785+0000 7f43405d3740 1 -- start start 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 Processor -- start 2024-04-22T06:37:28.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f433c902640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 0x7f43200a6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43377fe640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 0x7f43200a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f433c902640 1 -- 172.21.15.105:0/3141541876 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f432c0f2b40 con 0x7f4338101fe0 2024-04-22T06:37:28.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad8a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad812d0 0x56222ad91430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56222ad884b0 con 0x56222ad7fe40 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56222ac6d190 con 0x56222ad80830 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56222ad88630 con 0x56222ad812d0 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43405d3740 10 client.0 _ll_register_callbacks cb 0x56222ac688b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56502/0 (socket says 172.21.15.105:56502) 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 -- 172.21.15.105:0/1886252276 learned_addr learned my addr 172.21.15.105:0/1886252276 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43367fc640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad8a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43377fe640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad812d0 0x56222ad91430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 -- 172.21.15.105:0/1886252276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad812d0 msgr2=0x56222ad91430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad812d0 0x56222ad91430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 -- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 msgr2=0x56222ad8a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad8a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f43377fe640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 0x7f43200a6000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x56222ad7b200 tx=0x7f4330012820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 -- 172.21.15.105:0/1886252276 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222aac16b0 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4336ffd640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f43280116f0 tx=0x7f4328009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ecbbe55a96d41c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.786+0000 7f4316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4328029060 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f4328004c50 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4328012bc0 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4328016070 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f43405d3740 10 client.15327 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56222ad947a0 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.787+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4328012bc0 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f432802e070 con 0x56222ad7fe40 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 10 client.15327 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 20 client.15327 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ad969f0 0x56222ad97e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 4 client.15327 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56222acf2530 con 0x56222ad969f0 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43367fc640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ad969f0 0x56222ad97e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f4316ffd640 10 client.15327 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.788+0000 7f43367fc640 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ad969f0 0x56222ad97e60 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2247385bda6dc368 server_cookie=756915c9b6886b12 in_seq=0 out_seq=0 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f4316ffd640 1 -- 172.21.15.105:0/1886252276 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56222acf2530 con 0x56222ad969f0 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f4316ffd640 10 client.15327 handle_command_reply: tid=0 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 client.15327 shutdown 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 msgr2=0x56222ad89eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f43280116f0 tx=0x7f4328009d70 comp rx=0 tx=0).stop 2024-04-22T06:37:28.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 shutdown_connections 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ad969f0 0x56222ad97e60 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad812d0 0x56222ad91430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad8a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f4316ffd640 0 client.15327 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 --2- 172.21.15.105:0/1886252276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad7fe40 0x56222ad89eb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 >> 172.21.15.105:0/1886252276 conn(0x56222ad7bf40 msgr2=0x56222ad79a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 shutdown_connections 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 1 -- 172.21.15.105:0/1886252276 wait complete. 2024-04-22T06:37:28.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.789+0000 7f43405d3740 20 client.15327 trim_cache size 0 max 16384 2024-04-22T06:37:28.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 Processor -- start 2024-04-22T06:37:28.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 -- start start 2024-04-22T06:37:28.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad84840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad84d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56222ad961a0 con 0x56222ad812d0 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56222ad96870 con 0x56222ad80830 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43405d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56222ad96f40 con 0x56222ad7fe40 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43377fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43377fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56516/0 (socket says 172.21.15.105:56516) 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43377fe640 1 -- 172.21.15.105:0/3132872604 learned_addr learned my addr 172.21.15.105:0/3132872604 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.807+0000 7f43367fc640 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad84d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f4336ffd640 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad84840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 -- 172.21.15.105:0/3132872604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 msgr2=0x56222ad84840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad84840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 -- 172.21.15.105:0/3132872604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 msgr2=0x56222ad84d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad84d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 -- 172.21.15.105:0/3132872604 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56222ad7b630 con 0x56222ad812d0 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f43377fe640 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f43300206f0 tx=0x7f433001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd5e8a2e1e30319 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f431d7fa640 1 -- 172.21.15.105:0/3132872604 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330018040 con 0x56222ad812d0 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f431d7fa640 1 -- 172.21.15.105:0/3132872604 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433001a070 con 0x56222ad812d0 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.808+0000 7f431d7fa640 1 -- 172.21.15.105:0/3132872604 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433001e510 con 0x56222ad812d0 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- 172.21.15.105:0/3132872604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 msgr2=0x56222ad79170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f43300206f0 tx=0x7f433001d000 comp rx=0 tx=0).stop 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- 172.21.15.105:0/3132872604 shutdown_connections 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad812d0 0x56222ad79170 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad80830 0x56222ad84d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- 172.21.15.105:0/3132872604 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad84840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- 172.21.15.105:0/3132872604 >> 172.21.15.105:0/3132872604 conn(0x56222ad7bf40 msgr2=0x56222ad87eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- 172.21.15.105:0/3132872604 shutdown_connections 2024-04-22T06:37:28.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- 172.21.15.105:0/3132872604 wait complete. 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- start start 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 Processor -- start 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad83840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad81e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56222ad97ce0 con 0x56222ad80830 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56222ad97610 con 0x56222ad812d0 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56222ad983b0 con 0x56222ad7fe40 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43405d3740 10 client.0 _ll_register_callbacks cb 0x7f432c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43367fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56520/0 (socket says 172.21.15.105:56520) 2024-04-22T06:37:28.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f43367fc640 1 -- 172.21.15.105:0/63896656 learned_addr learned my addr 172.21.15.105:0/63896656 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.809+0000 7f4336ffd640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad83840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43377fe640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad81e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 -- 172.21.15.105:0/63896656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 msgr2=0x56222ad83840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad83840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 -- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 msgr2=0x56222ad81e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad81e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 -- 172.21.15.105:0/63896656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56222ad82570 con 0x56222ad80830 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43377fe640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad81e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43367fc640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f432c0bb320 tx=0x7f432c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4c43e0c0ca3c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0beec0 con 0x56222ad80830 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f432c0bd1c0 con 0x56222ad80830 2024-04-22T06:37:28.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f433001a070 con 0x56222ad80830 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f432c0c3270 con 0x56222ad80830 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 10 client.15333 fetch_fsmap learned FSMap version 26 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56222ae013a0 con 0x56222ad80830 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f433001a070 con 0x56222ad80830 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f432c0d81d0 con 0x56222ad80830 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 10 client.15333 fetch_fsmap finished waiting for FSMap version 26 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 10 client.15333 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 20 client.15333 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ae06050 0x56222ae04d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 4 client.15333 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.810+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "status"}) -- 0x56222acec3a0 con 0x56222ae06050 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4336ffd640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ae06050 0x56222ae04d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4316ffd640 10 client.15333 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:28.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4336ffd640 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ae06050 0x56222ae04d60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc3d96f28e25ee5 server_cookie=fe9fab744ee8047e in_seq=0 out_seq=0 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4316ffd640 1 -- 172.21.15.105:0/63896656 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56222acec3a0 con 0x56222ae06050 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4316ffd640 10 client.15333 handle_command_reply: tid=0 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 client.15333 shutdown 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 msgr2=0x56222ad83d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f432c0bb320 tx=0x7f432c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 shutdown_connections 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56222ae06050 0x56222ae04d60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56222ad812d0 0x56222ad81e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f4316ffd640 0 client.15333 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56222ad80830 0x56222ad83d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 --2- 172.21.15.105:0/63896656 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56222ad7fe40 0x56222ad83840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.811+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 >> 172.21.15.105:0/63896656 conn(0x56222ad7bf40 msgr2=0x56222ad79170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 shutdown_connections 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f43405d3740 1 -- 172.21.15.105:0/63896656 wait complete. 2024-04-22T06:37:28.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f43405d3740 20 client.15333 trim_cache size 0 max 16384 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 0, 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24500, 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 108.007413844, 2024-04-22T06:37:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 26, 2024-04-22T06:37:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 91, 2024-04-22T06:37:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 89, 2024-04-22T06:37:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 113.527648336 2024-04-22T06:37:28.819 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:28.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 msgr2=0x7f43200a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 0x7f43200a6000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x56222ad7b200 tx=0x7f4330012820 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 msgr2=0x7f4338198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f432c036040 tx=0x7f432c004040 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 shutdown_connections 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f43200a3b40 0x7f43200a6000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f43381029a0 0x7f433819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4338101fe0 0x7f4338198840 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 --2- 172.21.15.105:0/3141541876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4338100e90 0x7f4338198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 >> 172.21.15.105:0/3141541876 conn(0x7f43380fc400 msgr2=0x7f4338069170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 shutdown_connections 2024-04-22T06:37:28.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.812+0000 7f433d904640 1 -- 172.21.15.105:0/3141541876 wait complete. 2024-04-22T06:37:28.821 INFO:teuthology.run_tasks:Running task sequential... 2024-04-22T06:37:28.832 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-22T06:37:28.832 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-22T06:37:28.832 INFO:teuthology.task.print:Enabling mgr modules 2024-04-22T06:37:28.832 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-22T06:37:28.833 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:37:28.833 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:28.833 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-22T06:37:28.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- 172.21.15.105:0/456679962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c01020f0 msgr2=0x7f05c0102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/456679962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c01020f0 0x7f05c0102570 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009900 tx=0x7f05b002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- 172.21.15.105:0/456679962 shutdown_connections 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/456679962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c0102ab0 0x7f05c0109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/456679962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c01020f0 0x7f05c0102570 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/456679962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- 172.21.15.105:0/456679962 >> 172.21.15.105:0/456679962 conn(0x7f05c00fbbb0 msgr2=0x7f05c00fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- 172.21.15.105:0/456679962 shutdown_connections 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- 172.21.15.105:0/456679962 wait complete. 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 Processor -- start 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 -- start start 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.932+0000 7f05c7d8f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c01020f0 0x7f05c0197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05c010ca10 con 0x7f05c0102ab0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05c010c890 con 0x7f05c01013e0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05c010cb90 con 0x7f05c01020f0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56538/0 (socket says 172.21.15.105:56538) 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 -- 172.21.15.105:0/3217126987 learned_addr learned my addr 172.21.15.105:0/3217126987 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c5b2b640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c532a640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c01020f0 0x7f05c0197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 -- 172.21.15.105:0/3217126987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c01020f0 msgr2=0x7f05c0197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c01020f0 0x7f05c0197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 -- 172.21.15.105:0/3217126987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 msgr2=0x7f05c0196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 -- 172.21.15.105:0/3217126987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c019ebe0 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c5b2b640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c632c640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00eaa0 tx=0x7f05bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc00cdb0 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05bc005020 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c019eed0 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0116d0 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.933+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c019f390 con 0x7f05c0102ab0 2024-04-22T06:37:28.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.934+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c01081c0 con 0x7f05c0102ab0 2024-04-22T06:37:28.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.935+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f05bc0054a0 con 0x7f05c0102ab0 2024-04-22T06:37:28.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.936+0000 7f05b6ffd640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 0x7f05ac0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:28.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.936+0000 7f05c5b2b640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 0x7f05ac0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:28.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.936+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f05bc0d0920 con 0x7f05c0102ab0 2024-04-22T06:37:28.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.936+0000 7f05c5b2b640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 0x7f05ac0a6110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f05c006a370 tx=0x7f05b800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:28.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:28.940+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05bc016030 con 0x7f05c0102ab0 2024-04-22T06:37:29.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.168+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f05c002d810 con 0x7f05c0102ab0 2024-04-22T06:37:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1885582440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4114342686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1885582440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4114342686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1885582440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4114342686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:29.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.790+0000 7f05b6ffd640 1 -- 172.21.15.105:0/3217126987 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f05bc09d3a0 con 0x7f05c0102ab0 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 msgr2=0x7f05ac0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 0x7f05ac0a6110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f05c006a370 tx=0x7f05b800a5e0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 msgr2=0x7f05c019e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00eaa0 tx=0x7f05bc00ef60 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 shutdown_connections 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f05ac0a3c50 0x7f05ac0a6110 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05c0102ab0 0x7f05c019e570 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05c01020f0 0x7f05c0197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 --2- 172.21.15.105:0/3217126987 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05c01013e0 0x7f05c0196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 >> 172.21.15.105:0/3217126987 conn(0x7f05c00fbbb0 msgr2=0x7f05c0105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 shutdown_connections 2024-04-22T06:37:29.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.793+0000 7f05c7d8f640 1 -- 172.21.15.105:0/3217126987 wait complete. 2024-04-22T06:37:29.810 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 -- 172.21.15.105:0/3483524585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102050 msgr2=0x7f18b81024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/3483524585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102050 0x7f18b81024d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009900 tx=0x7f18a802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 -- 172.21.15.105:0/3483524585 shutdown_connections 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/3483524585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 0x7f18b8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/3483524585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102050 0x7f18b81024d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/3483524585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8100e70 0x7f18b8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 -- 172.21.15.105:0/3483524585 >> 172.21.15.105:0/3483524585 conn(0x7f18b80fc430 msgr2=0x7f18b80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 -- 172.21.15.105:0/3483524585 shutdown_connections 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.917+0000 7f18bfbb6640 1 -- 172.21.15.105:0/3483524585 wait complete. 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 Processor -- start 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 -- start start 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8102050 0x7f18b819f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 0x7f18b81a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810ca20 con 0x7f18b8100e70 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810c8a0 con 0x7f18b8102a10 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bfbb6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810cba0 con 0x7f18b8102050 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56568/0 (socket says 172.21.15.105:56568) 2024-04-22T06:37:29.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 -- 172.21.15.105:0/2435261836 learned_addr learned my addr 172.21.15.105:0/2435261836 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18be153640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 0x7f18b81a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd151640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8102050 0x7f18b819f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 -- 172.21.15.105:0/2435261836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8102050 msgr2=0x7f18b819f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8102050 0x7f18b819f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 -- 172.21.15.105:0/2435261836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 msgr2=0x7f18b81a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 0x7f18b81a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.918+0000 7f18bd952640 1 -- 172.21.15.105:0/2435261836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b81a1fd0 con 0x7f18b8100e70 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18bd952640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f18b400d950 tx=0x7f18b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4017070 con 0x7f18b8100e70 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b4009c30 con 0x7f18b8100e70 2024-04-22T06:37:29.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81a2260 con 0x7f18b8100e70 2024-04-22T06:37:29.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b400f620 con 0x7f18b8100e70 2024-04-22T06:37:29.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.919+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b81a2730 con 0x7f18b8100e70 2024-04-22T06:37:29.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.920+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b8108210 con 0x7f18b8100e70 2024-04-22T06:37:29.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.922+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f18b401a070 con 0x7f18b8100e70 2024-04-22T06:37:29.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.922+0000 7f18aeffd640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 0x7f18a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:29.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.922+0000 7f18bd151640 1 -- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 msgr2=0x7f18a40a6190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:29.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.922+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f18b40cfd30 con 0x7f18b8100e70 2024-04-22T06:37:29.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.922+0000 7f18bd151640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 0x7f18a40a6190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:37:29.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:29.927+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18b409c160 con 0x7f18b8100e70 2024-04-22T06:37:30.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.112+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f18b802d5c0 con 0x7f18b8100e70 2024-04-22T06:37:30.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.117+0000 7f18aeffd640 1 -- 172.21.15.105:0/2435261836 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f18b40a0460 con 0x7f18b8100e70 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 msgr2=0x7f18a40a6190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 0x7f18a40a6190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 msgr2=0x7f18b806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f18b400d950 tx=0x7f18b400de10 comp rx=0 tx=0).stop 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 shutdown_connections 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f18a40a3cd0 0x7f18a40a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18b8102a10 0x7f18b81a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18b8102050 0x7f18b819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 --2- 172.21.15.105:0/2435261836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8100e70 0x7f18b806ad40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 >> 172.21.15.105:0/2435261836 conn(0x7f18b80fc430 msgr2=0x7f18b81053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:30.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 shutdown_connections 2024-04-22T06:37:30.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.119+0000 7f18bfbb6640 1 -- 172.21.15.105:0/2435261836 wait complete. 2024-04-22T06:37:30.128 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:29 smithi105 ceph-mon[27916]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:37:30.128 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:29 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:37:30.128 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:29 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setuser ceph since I am not root 2024-04-22T06:37:30.128 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:29 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: ignoring --setgroup ceph since I am not root 2024-04-22T06:37:30.129 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:29 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:29.978+0000 7fb0cb6b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:37:30.129 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-22T06:37:30.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-mon[31346]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:37:30.152 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:37:30.152 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: ignoring --setuser ceph since I am not root 2024-04-22T06:37:30.153 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: ignoring --setgroup ceph since I am not root 2024-04-22T06:37:30.153 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-mgr[33214]: -- 172.21.15.177:0/4246858479 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0d06aa5a0 con 0x55f0d067ec00 2024-04-22T06:37:30.153 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:29.970+0000 7fb963c0e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:37:30.153 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:30.153+0000 7fb963c0e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-mon[31236]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: ignoring --setuser ceph since I am not root 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: ignoring --setgroup ceph since I am not root 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-mgr[32641]: -- 172.21.15.112:0/2012401206 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5613380d65a0 con 0x5613380aac00 2024-04-22T06:37:30.154 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:29.972+0000 7fc16f900200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:37:30.155 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:30.156+0000 7fc16f900200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 -- 172.21.15.105:0/750209038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 msgr2=0x7fcf4c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 --2- 172.21.15.105:0/750209038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c109320 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009ca0 tx=0x7fcf4402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 -- 172.21.15.105:0/750209038 shutdown_connections 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 --2- 172.21.15.105:0/750209038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c109320 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 --2- 172.21.15.105:0/750209038 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.232 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 --2- 172.21.15.105:0/750209038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 -- 172.21.15.105:0/750209038 >> 172.21.15.105:0/750209038 conn(0x7fcf4c0fc420 msgr2=0x7fcf4c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 -- 172.21.15.105:0/750209038 shutdown_connections 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.231+0000 7fcf510a5640 1 -- 172.21.15.105:0/750209038 wait complete. 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 Processor -- start 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 -- start start 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c06a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10ca00 con 0x7fcf4c102a00 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10c880 con 0x7fcf4c100e60 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.232+0000 7fcf510a5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10cb80 con 0x7fcf4c102040 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4b59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4b59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56584/0 (socket says 172.21.15.105:56584) 2024-04-22T06:37:30.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4b59e640 1 -- 172.21.15.105:0/2921448102 learned_addr learned my addr 172.21.15.105:0/2921448102 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c06a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4a59c640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 -- 172.21.15.105:0/2921448102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 msgr2=0x7fcf4c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 -- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 msgr2=0x7fcf4c19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 -- 172.21.15.105:0/2921448102 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c19fd40 con 0x7fcf4c100e60 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4b59e640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4a59c640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf4ad9d640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c06a190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e9c0 tx=0x7fcf3c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c00ccc0 con 0x7fcf4c100e60 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf3c0035f0 con 0x7fcf4c100e60 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c19ffd0 con 0x7fcf4c100e60 2024-04-22T06:37:30.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c00ce60 con 0x7fcf4c100e60 2024-04-22T06:37:30.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.233+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c1a04f0 con 0x7fcf4c100e60 2024-04-22T06:37:30.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.234+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c108200 con 0x7fcf4c100e60 2024-04-22T06:37:30.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.235+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7fcf3c004000 con 0x7fcf4c100e60 2024-04-22T06:37:30.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.236+0000 7fcf33fff640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 0x7fcf340a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:30.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.236+0000 7fcf4a59c640 1 -- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 msgr2=0x7fcf340a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:30.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.236+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fcf3c0d0990 con 0x7fcf4c100e60 2024-04-22T06:37:30.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.236+0000 7fcf4a59c640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 0x7fcf340a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:37:30.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.240+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf3c043e20 con 0x7fcf4c100e60 2024-04-22T06:37:30.418 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:30.171+0000 7fb0cb6b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:30.418 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:30.280+0000 7fb0cb6b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:37:30.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.425+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fcf4c062c60 con 0x7fcf4c100e60 2024-04-22T06:37:30.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.431+0000 7fcf33fff640 1 -- 172.21.15.105:0/2921448102 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7fcf3c09d3f0 con 0x7fcf4c100e60 2024-04-22T06:37:30.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 msgr2=0x7fcf340a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 0x7fcf340a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 msgr2=0x7fcf4c06a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c06a190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00e9c0 tx=0x7fcf3c00ee80 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 shutdown_connections 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fcf340a3ca0 0x7fcf340a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf4c102a00 0x7fcf4c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcf4c102040 0x7fcf4c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 --2- 172.21.15.105:0/2921448102 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf4c100e60 0x7fcf4c06a190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 >> 172.21.15.105:0/2921448102 conn(0x7fcf4c0fc420 msgr2=0x7fcf4c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 shutdown_connections 2024-04-22T06:37:30.433 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:30.434+0000 7fcf510a5640 1 -- 172.21.15.105:0/2921448102 wait complete. 2024-04-22T06:37:30.445 INFO:teuthology.run_tasks:Running task kclient... 2024-04-22T06:37:30.471 INFO:tasks.kclient:Mounting kernel clients... 2024-04-22T06:37:30.472 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-22T06:37:30.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:30.472 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-22T06:37:30.473 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:30.257+0000 7fc16f900200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:37:30.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:30.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link delete ceph-brx 2024-04-22T06:37:30.489 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:30.256+0000 7fb963c0e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:37:30.554 INFO:teuthology.orchestra.run.smithi105.stderr:Cannot find device "ceph-brx" 2024-04-22T06:37:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:37:30.556 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-22T06:37:30.556 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-22T06:37:30.556 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:30.556 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-22T06:37:30.700 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:30.612+0000 7fb0cb6b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:37:30.842 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:37:30.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:37:30.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-mon[31236]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-22T06:37:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2435261836' entity='client.admin' 2024-04-22T06:37:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-mon[31236]: from='client.? ' entity='client.admin' 2024-04-22T06:37:30.959 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:30.563+0000 7fc16f900200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:37:30.959 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:30.646+0000 7fc16f900200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:37:30.963 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:30.701+0000 7fb0cb6b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:37:30.964 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:37:30.964 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-mon[27916]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-22T06:37:30.964 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2435261836' entity='client.admin' 2024-04-22T06:37:30.964 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:30 smithi105 ceph-mon[27916]: from='client.? ' entity='client.admin' 2024-04-22T06:37:30.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3217126987' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-22T06:37:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-mon[31346]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-22T06:37:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2435261836' entity='client.admin' 2024-04-22T06:37:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-mon[31346]: from='client.? ' entity='client.admin' 2024-04-22T06:37:30.990 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:30.559+0000 7fb963c0e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:37:30.990 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:30.642+0000 7fb963c0e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:37:32.155 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:31.818+0000 7fc16f900200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:37:32.155 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:31.987+0000 7fc16f900200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:32.155 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:32.156+0000 7fc16f900200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:37:32.157 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:31.814+0000 7fb963c0e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:37:32.157 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:31.984+0000 7fb963c0e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:32.157 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:32.158+0000 7fb963c0e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:37:32.194 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:31 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:31.923+0000 7fb0cb6b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:37:32.194 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:32.104+0000 7fb0cb6b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:32.522 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:32.278+0000 7fb0cb6b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:37:32.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 -- 172.21.15.105:0/2242755846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec1019d0 msgr2=0x7f55ec10f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:32.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 --2- 172.21.15.105:0/2242755846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec1019d0 0x7f55ec10f910 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f55e400b0a0 tx=0x7f55e400b3b0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 -- 172.21.15.105:0/2242755846 shutdown_connections 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 --2- 172.21.15.105:0/2242755846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec108ad0 0x7f55ec108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 --2- 172.21.15.105:0/2242755846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec1019d0 0x7f55ec10f910 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 --2- 172.21.15.105:0/2242755846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec101030 0x7f55ec101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 -- 172.21.15.105:0/2242755846 >> 172.21.15.105:0/2242755846 conn(0x7f55ec077fb0 msgr2=0x7f55ec0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 -- 172.21.15.105:0/2242755846 shutdown_connections 2024-04-22T06:37:32.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.544+0000 7f55f2a09700 1 -- 172.21.15.105:0/2242755846 wait complete. 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 Processor -- start 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 -- start start 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 0x7f55ec1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec108ad0 0x7f55ec19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec112be0 con 0x7f55ec101030 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec112a60 con 0x7f55ec1019d0 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f2a09700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec112d60 con 0x7f55ec108ad0 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55ebfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55ebfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56594/0 (socket says 172.21.15.105:56594) 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55ebfff700 1 -- 172.21.15.105:0/1521605640 learned_addr learned my addr 172.21.15.105:0/1521605640 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55eb7fe700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 0x7f55ec1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:32.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.545+0000 7f55f0fa6700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec108ad0 0x7f55ec19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec108ad0 msgr2=0x7f55ec19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec108ad0 0x7f55ec19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 msgr2=0x7f55ec199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 -- 172.21.15.105:0/1521605640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec19e0a0 con 0x7f55ec1019d0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55ebfff700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55eb7fe700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 0x7f55ec1995d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00e9f0 tx=0x7f55dc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00cb70 con 0x7f55ec1019d0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec19e390 con 0x7f55ec1019d0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55dc00ee90 con 0x7f55ec1019d0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc003d40 con 0x7f55ec1019d0 2024-04-22T06:37:32.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.546+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec1a5c10 con 0x7f55ec1019d0 2024-04-22T06:37:32.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.547+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ec104350 con 0x7f55ec1019d0 2024-04-22T06:37:32.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.548+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f55dc003ee0 con 0x7f55ec1019d0 2024-04-22T06:37:32.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.549+0000 7f55e97fa700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 0x7f55d40a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:32.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.549+0000 7f55ebfff700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 msgr2=0x7f55d40a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:32.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.549+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f55dc0d04d0 con 0x7f55ec1019d0 2024-04-22T06:37:32.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.549+0000 7f55ebfff700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 0x7f55d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:37:32.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.553+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55dc09cf30 con 0x7f55ec1019d0 2024-04-22T06:37:32.708 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:32.443+0000 7fc16f900200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:37:32.708 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:32.536+0000 7fc16f900200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:37:32.739 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:32.446+0000 7fb963c0e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:37:32.740 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:32.538+0000 7fb963c0e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:37:32.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.750+0000 7f55ebfff700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 msgr2=0x7f55d40a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:32.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.750+0000 7f55ebfff700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 0x7f55d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:37:32.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.800+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f55ec02df80 con 0x7f55ec1019d0 2024-04-22T06:37:32.799 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:32.585+0000 7fb0cb6b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:37:32.800 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:32.680+0000 7fb0cb6b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:37:32.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.801+0000 7f55e97fa700 1 -- 172.21.15.105:0/1521605640 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f55dc09c830 con 0x7f55ec1019d0 2024-04-22T06:37:32.800 INFO:teuthology.orchestra.run.smithi105.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 msgr2=0x7f55d40a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 0x7f55d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 msgr2=0x7f55ec1995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 0x7f55ec1995d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00e9f0 tx=0x7f55dc00ed00 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 shutdown_connections 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f55d40a3c90 0x7f55d40a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55ec108ad0 0x7f55ec19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55ec101030 0x7f55ec199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 --2- 172.21.15.105:0/1521605640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55ec1019d0 0x7f55ec1995d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:32.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 >> 172.21.15.105:0/1521605640 conn(0x7f55ec077fb0 msgr2=0x7f55ec0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:32.803 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.803+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 shutdown_connections 2024-04-22T06:37:32.803 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:32.804+0000 7f55f2a09700 1 -- 172.21.15.105:0/1521605640 wait complete. 2024-04-22T06:37:33.091 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1521605640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:37:33.091 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:32.894+0000 7fb0cb6b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:37:33.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1521605640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:37:33.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:32.738+0000 7fc16f900200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:37:33.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:32.936+0000 7fc16f900200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:33.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1521605640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-22T06:37:33.240 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:32.744+0000 7fb963c0e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:37:33.240 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:32.942+0000 7fb963c0e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:33.418 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:33.092+0000 7fb0cb6b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi105.front.sepia.ceph.com 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-22T06:37:33.547 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:37:33.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:33.548 DEBUG:teuthology.orchestra.run.smithi105:> ip addr 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: inet 127.0.0.1/8 scope host lo 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 ::1/128 scope host 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b8 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f0 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b9 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:37:33.564 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f1 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:28 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:29 brd ff:ff:ff:ff:ff:ff 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: inet 172.21.15.105/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft 42113sec preferred_lft 42113sec 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 fe80::a72a:fe01:74f9:3d7c/64 scope link noprefixroute 2024-04-22T06:37:33.565 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-04-22T06:37:33.565 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add name ceph-brx type bridge 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr flush dev ceph-brx 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set ceph-brx up 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-22T06:37:33.566 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-22T06:37:33.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:33.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:33.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:33.750 DEBUG:teuthology.orchestra.run.smithi105:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-22T06:37:33.772 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:33.707+0000 7fb0cb6b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:37:33.774 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-04-22T06:37:33.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:33.776 DEBUG:teuthology.orchestra.run.smithi105:> route 2024-04-22T06:37:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:Kernel IP routing table 2024-04-22T06:37:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-22T06:37:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-22T06:37:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-22T06:37:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-22T06:37:33.851 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:37:33.852 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-22T06:37:33.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-22T06:37:33.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-22T06:37:33.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-22T06:37:33.852 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-22T06:37:33.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:33.947 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:33.531+0000 7fc16f900200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:37:33.963 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:33.544+0000 7fb963c0e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:37:34.121 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:34.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:34.127 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-22T06:37:34.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:34.143 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list-id 2024-04-22T06:37:34.198 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:37:34.198 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-22T06:37:34.198 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:37:34.198 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-22T06:37:34.198 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-22T06:37:34.208 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:33.949+0000 7fc16f900200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:37:34.209 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:34.045+0000 7fc16f900200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:37:34.239 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:33.965+0000 7fb963c0e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:37:34.239 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:34.060+0000 7fb963c0e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:37:34.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:34.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:34.169+0000 7fb0cb6b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:37:34.294 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:34.271+0000 7fb0cb6b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:37:34.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:34.315 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-22T06:37:34.315 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-22T06:37:34.316 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-22T06:37:34.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:34.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:34.434 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-22T06:37:34.434 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-04-22T06:37:34.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set brx.0 up 2024-04-22T06:37:34.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set dev brx.0 master ceph-brx 2024-04-22T06:37:34.434 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-04-22T06:37:34.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:34.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:34.510 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:37:34.511 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:34.534 INFO:teuthology.orchestra.run.smithi105.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-22T06:37:34.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-22T06:37:34.534 DEBUG:teuthology.orchestra.run.smithi105:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:34.589 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-22T06:37:34.589 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-22T06:37:34.589 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/c5087255-4608-4544-90e9-8fe59b3f44f2 2024-04-22T06:37:34.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:34.589 DEBUG:teuthology.orchestra.run.smithi105:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/c5087255-4608-4544-90e9-8fe59b3f44f2 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-22T06:37:34.650 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:34.494+0000 7fb0cb6b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:37:34.688 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-22T06:37:34.688 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-22T06:37:34.708 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:34.246+0000 7fc16f900200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:37:34.739 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:34.262+0000 7fb963c0e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:37:34.826 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-22T06:37:34.826 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-22T06:37:34.826 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.105:3300,172.21.15.112:3300,172.21.15.177:3300" 2024-04-22T06:37:34.826 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: trying mount with new device syntax: 0@48924e24-0071-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/c5087255-4608-4544-90e9-8fe59b3f44f2 2024-04-22T06:37:34.826 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.105:3300/172.21.15.112:3300/172.21.15.177:3300" will pass to kernel 2024-04-22T06:37:34.827 INFO:teuthology.orchestra.run.smithi105.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-22T06:37:34.827 INFO:teuthology.orchestra.run.smithi105.stdout: You just mounted a file system that supports labels which does not 2024-04-22T06:37:34.827 INFO:teuthology.orchestra.run.smithi105.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-22T06:37:34.828 INFO:teuthology.orchestra.run.smithi105.stdout: applications will generate AVC messages and not be allowed access to 2024-04-22T06:37:34.828 INFO:teuthology.orchestra.run.smithi105.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-22T06:37:34.829 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-22T06:37:34.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:34.829 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:34.898 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:37:34.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:37:34.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:34.995 INFO:teuthology.orchestra.run.smithi105.stdout:client15378 2024-04-22T06:37:34.995 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:37:34.995 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:37:35.011 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:37:35.095 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:34.765+0000 7fc16f900200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:37:35.095 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:34.856+0000 7fc16f900200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:37:35.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:37:35.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:35.107 INFO:teuthology.orchestra.run.smithi105.stdout:client15378 2024-04-22T06:37:35.108 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-22T06:37:35.120 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:34.786+0000 7fb963c0e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:37:35.120 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:34.878+0000 7fb963c0e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:37:35.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-22T06:37:35.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:35.165 INFO:teuthology.orchestra.run.smithi105.stdout:client15378 2024-04-22T06:37:35.165 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/48924e24-0071-11ef-bc93-c7b262605968.client15378/status') 2024-04-22T06:37:35.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-22T06:37:35.223 INFO:teuthology.orchestra.run.smithi105.stdout:instance: client.15378 (3)192.168.0.1:0/3678111489 2024-04-22T06:37:35.223 INFO:teuthology.orchestra.run.smithi105.stdout:blocklisted: false 2024-04-22T06:37:35.223 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records in 2024-04-22T06:37:35.223 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records out 2024-04-22T06:37:35.223 INFO:teuthology.orchestra.run.smithi105.stderr:70 bytes copied, 2.2659e-05 s, 3.1 MB/s 2024-04-22T06:37:35.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-22T06:37:35.230 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:37:35.240 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:37:35.240 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:35.241 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 -- 172.21.15.105:0/696688252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000102040 msgr2=0x7f10001024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 --2- 172.21.15.105:0/696688252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000102040 0x7f10001024c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009900 tx=0x7f0ff402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 -- 172.21.15.105:0/696688252 shutdown_connections 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 --2- 172.21.15.105:0/696688252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102a00 0x7f1000109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 --2- 172.21.15.105:0/696688252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000102040 0x7f10001024c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 --2- 172.21.15.105:0/696688252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000100e60 0x7f1000101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 -- 172.21.15.105:0/696688252 >> 172.21.15.105:0/696688252 conn(0x7f10000fc420 msgr2=0x7f10000fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 -- 172.21.15.105:0/696688252 shutdown_connections 2024-04-22T06:37:35.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.343+0000 7f1005dfb640 1 -- 172.21.15.105:0/696688252 wait complete. 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 Processor -- start 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 -- start start 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 0x7f1000198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000102a00 0x7f100019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f100010ca00 con 0x7f1000100e60 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f100010c880 con 0x7f1000102a00 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f1005dfb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f100010cb80 con 0x7f1000102040 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f0fff7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33836/0 (socket says 172.21.15.105:33836) 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f0fff7fe640 1 -- 172.21.15.105:0/3935730329 learned_addr learned my addr 172.21.15.105:0/3935730329 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f0ffeffd640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 0x7f1000198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.344+0000 7f0ffffff640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000102a00 0x7f100019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000102a00 msgr2=0x7f100019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000102a00 0x7f100019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 msgr2=0x7f1000198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 -- 172.21.15.105:0/3935730329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100019fd30 con 0x7f1000102040 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0fff7fe640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffeffd640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 0x7f1000198650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4043020 con 0x7f1000102040 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ff4004440 con 0x7f1000102040 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100019ffc0 con 0x7f1000102040 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4003c60 con 0x7f1000102040 2024-04-22T06:37:35.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.345+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001a0480 con 0x7f1000102040 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.346+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10001a0650 con 0x7f1000102040 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.347+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f0ff4046020 con 0x7f1000102040 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.348+0000 7f0ffcff9640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 0x7f0fe80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.348+0000 7f0fff7fe640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 msgr2=0x7f0fe80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.348+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7f0ff40f2a60 con 0x7f1000102040 2024-04-22T06:37:35.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.348+0000 7f0fff7fe640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 0x7f0fe80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:37:35.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.351+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ff404d020 con 0x7f1000102040 2024-04-22T06:37:35.412 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.047+0000 7fb0cb6b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:37:35.412 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.148+0000 7fb0cb6b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:37:35.435 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:35.096+0000 7fc16f900200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:37:35.435 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:35.228+0000 7fc16f900200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:37:35.436 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:35.322+0000 7fc16f900200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:37:35.436 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:35.436+0000 7fc16f900200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:35.466 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:35.121+0000 7fb963c0e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:37:35.466 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:35.255+0000 7fb963c0e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:37:35.466 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:35.351+0000 7fb963c0e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:37:35.466 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:35.467+0000 7fb963c0e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:35.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.549+0000 7f0fff7fe640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 msgr2=0x7f0fe80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:35.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.549+0000 7f0fff7fe640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 0x7f0fe80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:37:35.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.574+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f10001a0820 con 0x7f1000102040 2024-04-22T06:37:35.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.630+0000 7f0ffcff9640 1 -- 172.21.15.105:0/3935730329 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) ==== 118+0+0 (secure 0 0 0) 0x7f0ff40bf4e0 con 0x7f1000102040 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 msgr2=0x7f0fe80a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 0x7f0fe80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 msgr2=0x7f1000198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 0x7f1000198650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4000c00 tx=0x7f0ff4004040 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 shutdown_connections 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7f0fe80a3ca0 0x7f0fe80a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1000102a00 0x7f100019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1000102040 0x7f1000198650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 --2- 172.21.15.105:0/3935730329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1000100e60 0x7f1000198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 >> 172.21.15.105:0/3935730329 conn(0x7f10000fc420 msgr2=0x7f10000fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 shutdown_connections 2024-04-22T06:37:35.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.632+0000 7f1005dfb640 1 -- 172.21.15.105:0/3935730329 wait complete. 2024-04-22T06:37:35.642 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-22T06:37:35.662 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3935730329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.662 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-mon[27916]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.663 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.413+0000 7fb0cb6b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:37:35.663 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.552+0000 7fb0cb6b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:37:35.663 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.653+0000 7fb0cb6b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:37:35.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3935730329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-mon[31236]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3935730329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-mon[31346]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-22T06:37:35.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- 172.21.15.105:0/4292665267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc100e60 msgr2=0x7fbfdc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/4292665267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc100e60 0x7fbfdc101280 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009900 tx=0x7fbfcc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- 172.21.15.105:0/4292665267 shutdown_connections 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/4292665267 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc102a00 0x7fbfdc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/4292665267 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfdc102040 0x7fbfdc1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/4292665267 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc100e60 0x7fbfdc101280 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- 172.21.15.105:0/4292665267 >> 172.21.15.105:0/4292665267 conn(0x7fbfdc0fc420 msgr2=0x7fbfdc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- 172.21.15.105:0/4292665267 shutdown_connections 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- 172.21.15.105:0/4292665267 wait complete. 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 Processor -- start 2024-04-22T06:37:35.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.746+0000 7fbfe28fd640 1 -- start start 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc100e60 0x7fbfdc198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfdc102040 0x7fbfdc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 0x7fbfdc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbfdc10db30 con 0x7fbfdc102a00 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbfdc10d9b0 con 0x7fbfdc100e60 2024-04-22T06:37:35.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbfdc10dcb0 con 0x7fbfdc102040 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 0x7fbfdc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 0x7fbfdc19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33858/0 (socket says 172.21.15.105:33858) 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfdbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc100e60 0x7fbfdc198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 -- 172.21.15.105:0/3893137681 learned_addr learned my addr 172.21.15.105:0/3893137681 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfdb7fe640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfdc102040 0x7fbfdc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfdc102040 msgr2=0x7fbfdc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbfdc102040 0x7fbfdc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc100e60 msgr2=0x7fbfdc198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc100e60 0x7fbfdc198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfdc19fdb0 con 0x7fbfdc102a00 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbfdc100e60 0x7fbfdc198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe0e9a640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 0x7fbfdc19f740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400eb10 tx=0x7fbfd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4004b20 con 0x7fbfdc102a00 2024-04-22T06:37:35.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfd4004cc0 con 0x7fbfdc102a00 2024-04-22T06:37:35.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfdc1a00a0 con 0x7fbfdc102a00 2024-04-22T06:37:35.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4010670 con 0x7fbfdc102a00 2024-04-22T06:37:35.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.747+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfdc1a05a0 con 0x7fbfdc102a00 2024-04-22T06:37:35.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.748+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfdc108280 con 0x7fbfdc102a00 2024-04-22T06:37:35.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.750+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7fbfd4005000 con 0x7fbfdc102a00 2024-04-22T06:37:35.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.750+0000 7fbfd97fa640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 0x7fbfc40a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:35.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.750+0000 7fbfdbfff640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 msgr2=0x7fbfc40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:35.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.751+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8153+0+0 (secure 0 0 0) 0x7fbfd40d0bf0 con 0x7fbfdc102a00 2024-04-22T06:37:35.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.751+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 0x7fbfc40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:37:35.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.754+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfd4053ff0 con 0x7fbfdc102a00 2024-04-22T06:37:35.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:35.779+0000 7fb0cb6b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:37:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.952+0000 7fbfdbfff640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 msgr2=0x7fbfc40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:35.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.952+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 0x7fbfc40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:37:35.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:35.966+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fbfdc062390 con 0x7fbfc40a3ca0 2024-04-22T06:37:36.208 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:35.747+0000 7fc16f900200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:37:36.240 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:35.774+0000 7fb963c0e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:37:36.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.353+0000 7fbfdbfff640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 msgr2=0x7fbfc40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2556874834 2024-04-22T06:37:36.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.353+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 0x7fbfc40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:37:36.418 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:36.089+0000 7fb0cb6b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:37:36.631 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:36.372+0000 7fb963c0e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:37:36.631 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z[33210]: 2024-04-22T06:37:36.589+0000 7fb963c0e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:37:36.632 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:36.358+0000 7fc16f900200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:37:36.632 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y[32637]: 2024-04-22T06:37:36.577+0000 7fc16f900200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:37:36.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.645+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 20) ==== 138664+0+0 (secure 0 0 0) 0x7fbfd4099540 con 0x7fbfdc102a00 2024-04-22T06:37:36.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:36.700+0000 7fb0cb6b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:37:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:37:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:37:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: Standby manager daemon y restarted 2024-04-22T06:37:36.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: Standby manager daemon y started 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: Standby manager daemon z restarted 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: Standby manager daemon z started 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:37:36.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-mon[27916]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:36.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.935+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fbfd40ce770 con 0x7fbfdc102a00 2024-04-22T06:37:36.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.935+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 msgr2=0x7fbfc40a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-22T06:37:36.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:36.935+0000 7fbfd97fa640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/2556874834,v1:172.21.15.105:6801/2556874834] conn(0x7fbfc40a3ca0 0x7fbfc40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:36.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:37:36.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:37:36.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: Standby manager daemon y restarted 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: Standby manager daemon y started 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: Standby manager daemon z restarted 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: Standby manager daemon z started 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:37:36.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:36 smithi112 ceph-mon[31236]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: Standby manager daemon y restarted 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: Standby manager daemon y started 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.112:0/191663372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: Standby manager daemon z restarted 2024-04-22T06:37:36.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: Standby manager daemon z started 2024-04-22T06:37:36.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-22T06:37:36.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T06:37:36.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-22T06:37:36.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:36 smithi177 ceph-mon[31346]: from='mgr.? 172.21.15.177:0/685224383' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T06:37:37.173 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:36.922+0000 7fb0cb6b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.163+0000 7fb02e030700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.164+0000 7fb02e030700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.164+0000 7fb02e030700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.164+0000 7fb02e030700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.164+0000 7fb02e030700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.165+0000 7fb02afea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.165+0000 7fb02afea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.165+0000 7fb02afea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.165+0000 7fb02afea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.174 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.165+0000 7fb02afea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:37.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: Active manager daemon x restarted 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: Activating manager daemon x 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: mgrmap e21: x(active, starting, since 0.0122392s), standbys: y, z 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:37:37.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:37:37.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: Manager daemon x is now available 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:37:37.921 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:37:37.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.940+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mon.0 v2:172.21.15.105:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fbfd40ce460 con 0x7fbfdc102a00 2024-04-22T06:37:37.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.940+0000 7fbfd97fa640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbfc4035360 0x7fbfc4035740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:37.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.940+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fbfd4023e70 con 0x7fbfc4035360 2024-04-22T06:37:37.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.943+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbfc4035360 0x7fbfc4035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:37.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.944+0000 7fbfdbfff640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbfc4035360 0x7fbfc4035740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc005ab0 tx=0x7fbfcc005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: Active manager daemon x restarted 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: Activating manager daemon x 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: mgrmap e21: x(active, starting, since 0.0122392s), standbys: y, z 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:37.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:37:37.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:37:37.961 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: Manager daemon x is now available 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:37:37.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:37:37.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.970+0000 7fbfd97fa640 1 -- 172.21.15.105:0/3893137681 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fbfd4023e70 con 0x7fbfc4035360 2024-04-22T06:37:37.968 INFO:teuthology.orchestra.run.smithi105.stdout:{} 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbfc4035360 msgr2=0x7fbfc4035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbfc4035360 0x7fbfc4035740 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc005ab0 tx=0x7fbfcc005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 msgr2=0x7fbfdc19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 --2- 172.21.15.105:0/3893137681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbfdc102a00 0x7fbfdc19f740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400eb10 tx=0x7fbfd400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe0e9a640 1 -- 172.21.15.105:0/3893137681 reap_dead start 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 shutdown_connections 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 >> 172.21.15.105:0/3893137681 conn(0x7fbfdc0fc420 msgr2=0x7fbfdc105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 shutdown_connections 2024-04-22T06:37:37.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:37.972+0000 7fbfe28fd640 1 -- 172.21.15.105:0/3893137681 wait complete. 2024-04-22T06:37:37.983 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-22T06:37:37.994 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-22T06:37:38.011 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:38.012 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: mgrmap e20: x(active, since 8m), standbys: y, z 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: Active manager daemon x restarted 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: Activating manager daemon x 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: osdmap e92: 12 total, 12 up, 12 in 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: mgrmap e21: x(active, starting, since 0.0122392s), standbys: y, z 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T06:37:38.013 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:37:38.014 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T06:37:38.015 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T06:37:38.016 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: Manager daemon x is now available 2024-04-22T06:37:38.017 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:38.018 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-22T06:37:38.018 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-22T06:37:38.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- 172.21.15.105:0/163507127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 msgr2=0x7fd0ec1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 --2- 172.21.15.105:0/163507127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec1023c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009900 tx=0x7fd0dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- 172.21.15.105:0/163507127 shutdown_connections 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 --2- 172.21.15.105:0/163507127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 0x7fd0ec10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 --2- 172.21.15.105:0/163507127 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 0x7fd0ec103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 --2- 172.21.15.105:0/163507127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec1023c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- 172.21.15.105:0/163507127 >> 172.21.15.105:0/163507127 conn(0x7fd0ec0fd530 msgr2=0x7fd0ec0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- 172.21.15.105:0/163507127 shutdown_connections 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- 172.21.15.105:0/163507127 wait complete. 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 Processor -- start 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.125+0000 7fd0f2904640 1 -- start start 2024-04-22T06:37:38.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 0x7fd0ec1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 0x7fd0ec19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ec10db30 con 0x7fd0ec101fa0 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ec10d9b0 con 0x7fd0ec103b50 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ec10dcb0 con 0x7fd0ec103190 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34038/0 (socket says 172.21.15.105:34038) 2024-04-22T06:37:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 -- 172.21.15.105:0/3484107322 learned_addr learned my addr 172.21.15.105:0/3484107322 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0eb7fe640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 0x7fd0ec1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f0ea1640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 0x7fd0ec19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 -- 172.21.15.105:0/3484107322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 msgr2=0x7fd0ec1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 0x7fd0ec1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 -- 172.21.15.105:0/3484107322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 msgr2=0x7fd0ec19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 0x7fd0ec19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 -- 172.21.15.105:0/3484107322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ec19fdb0 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0ebfff640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc000c00 tx=0x7fd0dc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc043020 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0dc0044c0 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ec1a0040 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc003c90 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.126+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ec1a0590 con 0x7fd0ec101fa0 2024-04-22T06:37:38.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.127+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ec1092c0 con 0x7fd0ec101fa0 2024-04-22T06:37:38.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.128+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fd0dc046020 con 0x7fd0ec101fa0 2024-04-22T06:37:38.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.129+0000 7fd0e97fa640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 0x7fd0d40a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.129+0000 7fd0eb7fe640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 0x7fd0d40a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.129+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fd0dc0f26a0 con 0x7fd0ec101fa0 2024-04-22T06:37:38.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.130+0000 7fd0eb7fe640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 0x7fd0d40a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec1996a0 tx=0x7fd0e000b410 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.133+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0dc04d020 con 0x7fd0ec101fa0 2024-04-22T06:37:38.315 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.955+0000 7fb027fe4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:38.316 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.955+0000 7fb027fe4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:38.316 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.955+0000 7fb027fe4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:38.316 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.955+0000 7fb027fe4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:38.316 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:37:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:37:37.955+0000 7fb027fe4700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:37:38.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.329+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0ec062b50 con 0x7fd0ec101fa0 2024-04-22T06:37:38.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.331+0000 7fd0e97fa640 1 -- 172.21.15.105:0/3484107322 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7fd0dc0bf2c0 con 0x7fd0ec101fa0 2024-04-22T06:37:38.330 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:38.330 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 msgr2=0x7fd0d40a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 0x7fd0d40a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec1996a0 tx=0x7fd0e000b410 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 msgr2=0x7fd0ec198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc000c00 tx=0x7fd0dc0040c0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 shutdown_connections 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd0d40a39b0 0x7fd0d40a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0ec103b50 0x7fd0ec19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd0ec103190 0x7fd0ec1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 --2- 172.21.15.105:0/3484107322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0ec101fa0 0x7fd0ec198190 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 >> 172.21.15.105:0/3484107322 conn(0x7fd0ec0fd530 msgr2=0x7fd0ec106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 shutdown_connections 2024-04-22T06:37:38.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.333+0000 7fd0f2904640 1 -- 172.21.15.105:0/3484107322 wait complete. 2024-04-22T06:37:38.343 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:38.343 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:38.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 -- 172.21.15.105:0/209588139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 msgr2=0x7f74ac101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 --2- 172.21.15.105:0/209588139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac101280 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f749c009900 tx=0x7f749c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 -- 172.21.15.105:0/209588139 shutdown_connections 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 --2- 172.21.15.105:0/209588139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 --2- 172.21.15.105:0/209588139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 --2- 172.21.15.105:0/209588139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac101280 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 -- 172.21.15.105:0/209588139 >> 172.21.15.105:0/209588139 conn(0x7f74ac0fc420 msgr2=0x7f74ac0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 -- 172.21.15.105:0/209588139 shutdown_connections 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.448+0000 7f74b214b640 1 -- 172.21.15.105:0/209588139 wait complete. 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 Processor -- start 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 -- start start 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac10cab0 con 0x7f74ac100e60 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac10c930 con 0x7f74ac102a00 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74b214b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac10cc30 con 0x7f74ac102040 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74ab7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34066/0 (socket says 172.21.15.105:34066) 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74ab7fe640 1 -- 172.21.15.105:0/3166428615 learned_addr learned my addr 172.21.15.105:0/3166428615 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.449+0000 7f74abfff640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74aaffd640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 -- 172.21.15.105:0/3166428615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 msgr2=0x7f74ac1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 -- 172.21.15.105:0/3166428615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 msgr2=0x7f74ac19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 -- 172.21.15.105:0/3166428615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ac19fe90 con 0x7f74ac100e60 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74abfff640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74aaffd640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:38.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74ab7fe640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c043020 con 0x7f74ac100e60 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ac1a0120 con 0x7f74ac100e60 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f749c0044c0 con 0x7f74ac100e60 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c003c60 con 0x7f74ac100e60 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.450+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ac1a06f0 con 0x7f74ac100e60 2024-04-22T06:37:38.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.451+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ac1a08f0 con 0x7f74ac100e60 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.452+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f749c046020 con 0x7f74ac100e60 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.452+0000 7f74a8ff9640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 0x7f74940a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.453+0000 7f74aaffd640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 0x7f74940a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.453+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f749c0f2400 con 0x7f74ac100e60 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.453+0000 7f74aaffd640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 0x7f74940a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74a0006fd0 tx=0x7f74a0008040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.457+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f749c04d020 con 0x7f74ac100e60 2024-04-22T06:37:38.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.642+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74ac1a0b60 con 0x7f74ac100e60 2024-04-22T06:37:38.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.642+0000 7f74a8ff9640 1 -- 172.21.15.105:0/3166428615 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f749c03c350 con 0x7f74ac100e60 2024-04-22T06:37:38.641 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:38.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.644+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 msgr2=0x7f74940a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.644+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 0x7f74940a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74a0006fd0 tx=0x7f74a0008040 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 msgr2=0x7f74ac198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f749c000c00 tx=0x7f749c0040c0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 shutdown_connections 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f74940a39b0 0x7f74940a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f74ac102a00 0x7f74ac19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f74ac102040 0x7f74ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 --2- 172.21.15.105:0/3166428615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f74ac100e60 0x7f74ac198270 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 >> 172.21.15.105:0/3166428615 conn(0x7f74ac0fc420 msgr2=0x7f74ac06a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 shutdown_connections 2024-04-22T06:37:38.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.645+0000 7f74b214b640 1 -- 172.21.15.105:0/3166428615 wait complete. 2024-04-22T06:37:38.655 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- 172.21.15.105:0/3037600408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4101fa0 msgr2=0x7f62c41023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- 172.21.15.105:0/3037600408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4101fa0 0x7f62c41023c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009900 tx=0x7f62b402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- 172.21.15.105:0/3037600408 shutdown_connections 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- 172.21.15.105:0/3037600408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4103b50 0x7f62c410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- 172.21.15.105:0/3037600408 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 0x7f62c4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- 172.21.15.105:0/3037600408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4101fa0 0x7f62c41023c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- 172.21.15.105:0/3037600408 >> 172.21.15.105:0/3037600408 conn(0x7f62c40fd530 msgr2=0x7f62c40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- 172.21.15.105:0/3037600408 shutdown_connections 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- 172.21.15.105:0/3037600408 wait complete. 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 Processor -- start 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 -- start start 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4101fa0 0x7f62c4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 0x7f62c4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.765+0000 7f62cb384640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62cb384640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410dae0 con 0x7f62c4103b50 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62cb384640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410d960 con 0x7f62c4101fa0 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62cb384640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410dc60 con 0x7f62c4103190 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34092/0 (socket says 172.21.15.105:34092) 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 -- 172.21.15.105:0/3596918810 learned_addr learned my addr 172.21.15.105:0/3596918810 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:38.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c891f640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 0x7f62c4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9120640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4101fa0 0x7f62c4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 -- 172.21.15.105:0/3596918810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 msgr2=0x7f62c4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 0x7f62c4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 -- 172.21.15.105:0/3596918810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4101fa0 msgr2=0x7f62c4075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4101fa0 0x7f62c4075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 -- 172.21.15.105:0/3596918810 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c4075110 con 0x7f62c4103b50 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62c9921640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00eb10 tx=0x7f62bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc004b20 con 0x7f62c4103b50 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c41a2300 con 0x7f62c4103b50 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62bc004cc0 con 0x7f62c4103b50 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc010670 con 0x7f62c4103b50 2024-04-22T06:37:38.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.766+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c41a2800 con 0x7f62c4103b50 2024-04-22T06:37:38.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.768+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c41093d0 con 0x7f62c4103b50 2024-04-22T06:37:38.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.768+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f62bc005020 con 0x7f62c4103b50 2024-04-22T06:37:38.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.769+0000 7f62ba7fc640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 0x7f62b00a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:38.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.769+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f62bc0d0620 con 0x7f62c4103b50 2024-04-22T06:37:38.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.769+0000 7f62c9120640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 0x7f62b00a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:38.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.770+0000 7f62c9120640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 0x7f62b00a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005ab0 tx=0x7f62b4005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:38.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.774+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62bc09d240 con 0x7f62c4103b50 2024-04-22T06:37:38.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.977+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f62c4062b50 con 0x7f62c4103b50 2024-04-22T06:37:38.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.978+0000 7f62ba7fc640 1 -- 172.21.15.105:0/3596918810 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f62bc09cb40 con 0x7f62c4103b50 2024-04-22T06:37:38.976 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:38.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 msgr2=0x7f62b00a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 0x7f62b00a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005ab0 tx=0x7f62b4005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 msgr2=0x7f62c4074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00eb10 tx=0x7f62bc00c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 shutdown_connections 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62b00a39b0 0x7f62b00a5e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62c4103b50 0x7f62c4074880 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62c4103190 0x7f62c4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 --2- 172.21.15.105:0/3596918810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c4101fa0 0x7f62c4075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 >> 172.21.15.105:0/3596918810 conn(0x7f62c40fd530 msgr2=0x7f62c41065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 shutdown_connections 2024-04-22T06:37:38.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:38.980+0000 7f62cb384640 1 -- 172.21.15.105:0/3596918810 wait complete. 2024-04-22T06:37:38.991 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:39.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 -- 172.21.15.105:0/1336420475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 msgr2=0x7f6460104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 --2- 172.21.15.105:0/1336420475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460104ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6458009900 tx=0x7f645802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:39.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 -- 172.21.15.105:0/1336420475 shutdown_connections 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 --2- 172.21.15.105:0/1336420475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460107d30 0x7f646010a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 --2- 172.21.15.105:0/1336420475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460105400 0x7f64601077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 --2- 172.21.15.105:0/1336420475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460104ec0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.109+0000 7f6464a2e640 1 -- 172.21.15.105:0/1336420475 >> 172.21.15.105:0/1336420475 conn(0x7f64600fc400 msgr2=0x7f64600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- 172.21.15.105:0/1336420475 shutdown_connections 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- 172.21.15.105:0/1336420475 wait complete. 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 Processor -- start 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- start start 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 0x7f6460198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 0x7f646019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f646010dae0 con 0x7f6460102ad0 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f646010d960 con 0x7f6460105400 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.110+0000 7f6464a2e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f646010dc60 con 0x7f6460107d30 2024-04-22T06:37:39.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34134/0 (socket says 172.21.15.105:34134) 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 0x7f6460198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 -- 172.21.15.105:0/2339047144 learned_addr learned my addr 172.21.15.105:0/2339047144 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645ed9d640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 0x7f646019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 -- 172.21.15.105:0/2339047144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 msgr2=0x7f646019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 0x7f646019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 -- 172.21.15.105:0/2339047144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 msgr2=0x7f6460198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 0x7f6460198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 -- 172.21.15.105:0/2339047144 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646019fe60 con 0x7f6460102ad0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645dd9b640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 0x7f6460198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645ed9d640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 0x7f646019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f645e59c640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6458006010 tx=0x7f6458005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458043020 con 0x7f6460102ad0 2024-04-22T06:37:39.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64580043d0 con 0x7f6460102ad0 2024-04-22T06:37:39.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64601a00f0 con 0x7f6460102ad0 2024-04-22T06:37:39.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.111+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458003b80 con 0x7f6460102ad0 2024-04-22T06:37:39.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.112+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64601a0690 con 0x7f6460102ad0 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.113+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64600fdef0 con 0x7f6460102ad0 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.114+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f6458046020 con 0x7f6460102ad0 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.114+0000 7f644f7fe640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 0x7f64480a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.114+0000 7f645dd9b640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 0x7f64480a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.114+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f64580f2880 con 0x7f6460102ad0 2024-04-22T06:37:39.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.115+0000 7f645dd9b640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 0x7f64480a5ef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6460199750 tx=0x7f6454009500 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.118+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f645804d020 con 0x7f6460102ad0 2024-04-22T06:37:39.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: mgrmap e22: x(active, since 1.01963s), standbys: y, z 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: [22/Apr/2024:06:37:38] ENGINE Bus STARTING 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: [22/Apr/2024:06:37:38] ENGINE Serving on http://172.21.15.105:8765 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: [22/Apr/2024:06:37:38] ENGINE Serving on https://172.21.15.105:7150 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: [22/Apr/2024:06:37:38] ENGINE Bus STARTED 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3484107322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3166428615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:38 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: mgrmap e22: x(active, since 1.01963s), standbys: y, z 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: [22/Apr/2024:06:37:38] ENGINE Bus STARTING 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: [22/Apr/2024:06:37:38] ENGINE Serving on http://172.21.15.105:8765 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: [22/Apr/2024:06:37:38] ENGINE Serving on https://172.21.15.105:7150 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: [22/Apr/2024:06:37:38] ENGINE Bus STARTED 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3484107322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3166428615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:38 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.312+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6460062b50 con 0x7f6460102ad0 2024-04-22T06:37:39.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.312+0000 7f644f7fe640 1 -- 172.21.15.105:0/2339047144 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f64580bf4a0 con 0x7f6460102ad0 2024-04-22T06:37:39.311 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 msgr2=0x7f64480a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 0x7f64480a5ef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6460199750 tx=0x7f6454009500 comp rx=0 tx=0).stop 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 msgr2=0x7f6460198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6458006010 tx=0x7f6458005e40 comp rx=0 tx=0).stop 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 shutdown_connections 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64480a3a30 0x7f64480a5ef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6460107d30 0x7f646019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6460105400 0x7f6460198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 --2- 172.21.15.105:0/2339047144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6460102ad0 0x7f6460198240 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 >> 172.21.15.105:0/2339047144 conn(0x7f64600fc400 msgr2=0x7f64601035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 shutdown_connections 2024-04-22T06:37:39.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.314+0000 7f6464a2e640 1 -- 172.21.15.105:0/2339047144 wait complete. 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: mgrmap e22: x(active, since 1.01963s), standbys: y, z 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: [22/Apr/2024:06:37:38] ENGINE Bus STARTING 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: [22/Apr/2024:06:37:38] ENGINE Serving on http://172.21.15.105:8765 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: [22/Apr/2024:06:37:38] ENGINE Serving on https://172.21.15.105:7150 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: [22/Apr/2024:06:37:38] ENGINE Bus STARTED 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3484107322' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:39.323 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3166428615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:39.324 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.324 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:38 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:39.325 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:39.427 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- 172.21.15.105:0/2037070639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc107c50 msgr2=0x7fdabc10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.427 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 --2- 172.21.15.105:0/2037070639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc107c50 0x7fdabc10a0a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009ca0 tx=0x7fdab402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.427 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- 172.21.15.105:0/2037070639 shutdown_connections 2024-04-22T06:37:39.427 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 --2- 172.21.15.105:0/2037070639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc107c50 0x7fdabc10a0a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 --2- 172.21.15.105:0/2037070639 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 0x7fdabc107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 --2- 172.21.15.105:0/2037070639 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc1029f0 0x7fdabc104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- 172.21.15.105:0/2037070639 >> 172.21.15.105:0/2037070639 conn(0x7fdabc0fc3e0 msgr2=0x7fdabc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- 172.21.15.105:0/2037070639 shutdown_connections 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- 172.21.15.105:0/2037070639 wait complete. 2024-04-22T06:37:39.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 Processor -- start 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.426+0000 7fdac2515640 1 -- start start 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 0x7fdabc1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc107c50 0x7fdabc100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10dab0 con 0x7fdabc1029f0 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10d930 con 0x7fdabc107c50 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac2515640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc10dc30 con 0x7fdabc105320 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34148/0 (socket says 172.21.15.105:34148) 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 -- 172.21.15.105:0/2633673869 learned_addr learned my addr 172.21.15.105:0/2633673869 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:39.429 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabb7fe640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 0x7fdabc1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdac0ab2640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc107c50 0x7fdabc100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 -- 172.21.15.105:0/2633673869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 msgr2=0x7fdabc1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 0x7fdabc1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 -- 172.21.15.105:0/2633673869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc107c50 msgr2=0x7fdabc100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc107c50 0x7fdabc100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 -- 172.21.15.105:0/2633673869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc101770 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.427+0000 7fdabbfff640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00e9f0 tx=0x7fdaac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.428+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac00cd80 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.428+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1a2270 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.428+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaac0108e0 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.428+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac018980 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.428+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc1a2810 con 0x7fdabc1029f0 2024-04-22T06:37:39.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.429+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc0fded0 con 0x7fdabc1029f0 2024-04-22T06:37:39.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.430+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fdaac010430 con 0x7fdabc1029f0 2024-04-22T06:37:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.431+0000 7fdab97fa640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 0x7fdaa40a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.431+0000 7fdabb7fe640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 0x7fdaa40a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.431+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fdaac0d0860 con 0x7fdabc1029f0 2024-04-22T06:37:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.432+0000 7fdabb7fe640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 0x7fdaa40a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdabc1020d0 tx=0x7fdab0005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.434+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdaac09ce30 con 0x7fdabc1029f0 2024-04-22T06:37:39.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.624+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdabc062b50 con 0x7fdabc1029f0 2024-04-22T06:37:39.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.625+0000 7fdab97fa640 1 -- 172.21.15.105:0/2633673869 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7fdaac0a1130 con 0x7fdabc1029f0 2024-04-22T06:37:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:39.624 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:39.626 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 msgr2=0x7fdaa40a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.626 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 0x7fdaa40a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdabc1020d0 tx=0x7fdab0005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 msgr2=0x7fdabc102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00e9f0 tx=0x7fdaac00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 shutdown_connections 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdaa40a39b0 0x7fdaa40a5e70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdabc107c50 0x7fdabc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdabc105320 0x7fdabc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 --2- 172.21.15.105:0/2633673869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdabc1029f0 0x7fdabc102360 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 >> 172.21.15.105:0/2633673869 conn(0x7fdabc0fc3e0 msgr2=0x7fdabc1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 shutdown_connections 2024-04-22T06:37:39.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.628+0000 7fdac2515640 1 -- 172.21.15.105:0/2633673869 wait complete. 2024-04-22T06:37:39.638 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:39.639 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1585271592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018102040 msgr2=0x7ff0181024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1585271592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018102040 0x7ff0181024c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff008009900 tx=0x7ff00802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1585271592 shutdown_connections 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1585271592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102a00 0x7ff018109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1585271592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018102040 0x7ff0181024c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1585271592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018100e60 0x7ff018101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1585271592 >> 172.21.15.105:0/1585271592 conn(0x7ff0180fc440 msgr2=0x7ff0180fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1585271592 shutdown_connections 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.744+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1585271592 wait complete. 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 Processor -- start 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 -- start start 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102040 0x7ff01819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 0x7ff01819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff01810ca10 con 0x7ff018100e60 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff01810c890 con 0x7ff018102040 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01fd8f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff01810cb90 con 0x7ff018102a00 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34162/0 (socket says 172.21.15.105:34162) 2024-04-22T06:37:39.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 -- 172.21.15.105:0/1135122535 learned_addr learned my addr 172.21.15.105:0/1135122535 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01d32a640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102040 0x7ff01819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01e32c640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 0x7ff01819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 -- 172.21.15.105:0/1135122535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 msgr2=0x7ff01819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 0x7ff01819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 -- 172.21.15.105:0/1135122535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102040 msgr2=0x7ff01819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102040 0x7ff01819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01db2b640 1 -- 172.21.15.105:0/1135122535 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01819feb0 con 0x7ff018100e60 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.745+0000 7ff01e32c640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 0x7ff01819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff01db2b640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff01400e9c0 tx=0x7ff01400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01400cd80 con 0x7ff018100e60 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0181a0140 con 0x7ff018100e60 2024-04-22T06:37:39.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0140035f0 con 0x7ff018100e60 2024-04-22T06:37:39.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014010690 con 0x7ff018100e60 2024-04-22T06:37:39.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.746+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0181a0660 con 0x7ff018100e60 2024-04-22T06:37:39.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.747+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff018108320 con 0x7ff018100e60 2024-04-22T06:37:39.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.748+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7ff014004000 con 0x7ff018100e60 2024-04-22T06:37:39.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.748+0000 7ff00effd640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 0x7ff0040a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:39.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.748+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7ff0140d0650 con 0x7ff018100e60 2024-04-22T06:37:39.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.748+0000 7ff01d32a640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 0x7ff0040a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:39.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.749+0000 7ff01d32a640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 0x7ff0040a5e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff008009230 tx=0x7ff008005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:39.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.753+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff01409d270 con 0x7ff018100e60 2024-04-22T06:37:39.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.915+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 23) ==== 138818+0+0 (secure 0 0 0) 0x7ff0140991e0 con 0x7ff018100e60 2024-04-22T06:37:39.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.946+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff01802d5c0 con 0x7ff018100e60 2024-04-22T06:37:39.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.950+0000 7ff00effd640 1 -- 172.21.15.105:0/1135122535 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7ff01802d5c0 con 0x7ff018100e60 2024-04-22T06:37:39.948 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 msgr2=0x7ff0040a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 0x7ff0040a5e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff008009230 tx=0x7ff008005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 msgr2=0x7ff01806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff01400e9c0 tx=0x7ff01400ee80 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 shutdown_connections 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff0040a39b0 0x7ff0040a5e70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff018102a00 0x7ff01819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff018102040 0x7ff01819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 --2- 172.21.15.105:0/1135122535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff018100e60 0x7ff01806b0c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 >> 172.21.15.105:0/1135122535 conn(0x7ff0180fc440 msgr2=0x7ff0181054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 shutdown_connections 2024-04-22T06:37:39.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:39.952+0000 7ff01fd8f640 1 -- 172.21.15.105:0/1135122535 wait complete. 2024-04-22T06:37:39.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:40.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3669552075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 msgr2=0x7f1db0103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3669552075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0103740 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009900 tx=0x7f1da002bc80 comp rx=0 tx=0).stop 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3669552075 shutdown_connections 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3669552075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 0x7f1db010a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3669552075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0103740 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3669552075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3669552075 >> 172.21.15.105:0/3669552075 conn(0x7f1db00fbb80 msgr2=0x7f1db00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3669552075 shutdown_connections 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3669552075 wait complete. 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 Processor -- start 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.070+0000 7f1db7cd9640 1 -- start start 2024-04-22T06:37:40.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 0x7f1db019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1db010daf0 con 0x7f1db01032c0 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1db010d970 con 0x7f1db0103c80 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db7cd9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1db010dc70 con 0x7f1db0100250 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5274640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5274640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34174/0 (socket says 172.21.15.105:34174) 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5274640 1 -- 172.21.15.105:0/3452612331 learned_addr learned my addr 172.21.15.105:0/3452612331 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db6276640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 0x7f1db019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 -- 172.21.15.105:0/3452612331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 msgr2=0x7f1db019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 0x7f1db019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.072 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 -- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 msgr2=0x7f1db0198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 -- 172.21.15.105:0/3452612331 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db019fe30 con 0x7f1db0100250 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.071+0000 7f1db5a75640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0198210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00e9c0 tx=0x7f1dac00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.072+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac00ccc0 con 0x7f1db0100250 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.072+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db01a0120 con 0x7f1db0100250 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.072+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db01a0670 con 0x7f1db0100250 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.072+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dac0035f0 con 0x7f1db0100250 2024-04-22T06:37:40.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.072+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dac00ce60 con 0x7f1db0100250 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.073+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db0109510 con 0x7f1db0100250 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.073+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 23) ==== 138818+0+0 (secure 0 0 0) 0x7f1dac003a80 con 0x7f1db0100250 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.074+0000 7f1da6ffd640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 0x7f1d9c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.074+0000 7f1db5274640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 0x7f1d9c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.074+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f1dac0d0db0 con 0x7f1db0100250 2024-04-22T06:37:40.074 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.074+0000 7f1db5274640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 0x7f1d9c0a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1db0199720 tx=0x7f1da002c7d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.079+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1dac09d350 con 0x7f1db0100250 2024-04-22T06:37:40.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:40.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3596918810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2339047144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2633673869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:40.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:39 smithi112 ceph-mon[31236]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3596918810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2339047144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2633673869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:40.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:39 smithi177 ceph-mon[31346]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:37:40.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.286+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1db002d5c0 con 0x7f1db0100250 2024-04-22T06:37:40.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.286+0000 7f1da6ffd640 1 -- 172.21.15.105:0/3452612331 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f1dac09cc50 con 0x7f1db0100250 2024-04-22T06:37:40.285 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 msgr2=0x7f1d9c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 0x7f1d9c0a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1db0199720 tx=0x7f1da002c7d0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 msgr2=0x7f1db0198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0198210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00e9c0 tx=0x7f1dac00ee80 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 shutdown_connections 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1d9c0a3db0 0x7f1d9c0a6270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1db0103c80 0x7f1db019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1db01032c0 0x7f1db0198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 --2- 172.21.15.105:0/3452612331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1db0100250 0x7f1db0198210 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 >> 172.21.15.105:0/3452612331 conn(0x7f1db00fbb80 msgr2=0x7f1db01066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 shutdown_connections 2024-04-22T06:37:40.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.288+0000 7f1db7cd9640 1 -- 172.21.15.105:0/3452612331 wait complete. 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3596918810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:40.297 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2339047144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:40.298 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2633673869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:40.298 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:39 smithi105 ceph-mon[27916]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-22T06:37:40.299 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:40.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2859561629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 msgr2=0x7f3a14107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2859561629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14107750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009860 tx=0x7f3a0002bb70 comp rx=0 tx=0).stop 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2859561629 shutdown_connections 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2859561629 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2859561629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14107750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2859561629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2859561629 >> 172.21.15.105:0/2859561629 conn(0x7f3a140fc440 msgr2=0x7f3a140fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2859561629 shutdown_connections 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.404+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2859561629 wait complete. 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 Processor -- start 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 -- start start 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1410daa0 con 0x7f3a14107c90 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1410d920 con 0x7f3a14105360 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a1b1f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1410dc20 con 0x7f3a14102a30 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a18f90640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34184/0 (socket says 172.21.15.105:34184) 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 -- 172.21.15.105:0/2793154243 learned_addr learned my addr 172.21.15.105:0/2793154243 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a18f90640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53756/0 (socket says 172.21.15.105:53756) 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a13fff640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.407 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 -- 172.21.15.105:0/2793154243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 msgr2=0x7f3a14198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 -- 172.21.15.105:0/2793154243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 msgr2=0x7f3a14198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 -- 172.21.15.105:0/2793154243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1419fe40 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.405+0000 7f3a19791640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009fd0 tx=0x7f3a0c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.406+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004b20 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.406+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c003e20 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.406+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a141a0130 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.406+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004820 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.406+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a141a0650 con 0x7f3a14107c90 2024-04-22T06:37:40.408 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.407+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a140fdf30 con 0x7f3a14107c90 2024-04-22T06:37:40.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.408+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 138818+0+0 (secure 0 0 0) 0x7f3a0c01d020 con 0x7f3a14107c90 2024-04-22T06:37:40.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.409+0000 7f3a11ffb640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 0x7f3a040a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.409+0000 7f3a18f90640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 0x7f3a040a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.409+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f3a0c0d08f0 con 0x7f3a14107c90 2024-04-22T06:37:40.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.409+0000 7f3a18f90640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 0x7f3a040a62f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005fd0 tx=0x7f3a0800b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.413+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a0c09ce90 con 0x7f3a14107c90 2024-04-22T06:37:40.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.600+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3a14062b50 con 0x7f3a14107c90 2024-04-22T06:37:40.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.600+0000 7f3a11ffb640 1 -- 172.21.15.105:0/2793154243 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f3a0c09c790 con 0x7f3a14107c90 2024-04-22T06:37:40.599 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 msgr2=0x7f3a040a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 0x7f3a040a62f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a08005fd0 tx=0x7f3a0800b390 comp rx=0 tx=0).stop 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 msgr2=0x7f3a1419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009fd0 tx=0x7f3a0c00c800 comp rx=0 tx=0).stop 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 shutdown_connections 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3a040a3e30 0x7f3a040a62f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a14107c90 0x7f3a1419f7d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a14105360 0x7f3a14198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 --2- 172.21.15.105:0/2793154243 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3a14102a30 0x7f3a14198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 >> 172.21.15.105:0/2793154243 conn(0x7f3a140fc440 msgr2=0x7f3a141048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 shutdown_connections 2024-04-22T06:37:40.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.602+0000 7f3a1b1f4640 1 -- 172.21.15.105:0/2793154243 wait complete. 2024-04-22T06:37:40.612 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-22T06:37:40.612 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:40.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- 172.21.15.105:0/1882510742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103b30 msgr2=0x7f8ba810a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- 172.21.15.105:0/1882510742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103b30 0x7f8ba810a3c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009ca0 tx=0x7f8ba002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- 172.21.15.105:0/1882510742 shutdown_connections 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- 172.21.15.105:0/1882510742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103b30 0x7f8ba810a3c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- 172.21.15.105:0/1882510742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8103170 0x7f8ba81035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- 172.21.15.105:0/1882510742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8101f80 0x7f8ba81023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- 172.21.15.105:0/1882510742 >> 172.21.15.105:0/1882510742 conn(0x7f8ba80fd530 msgr2=0x7f8ba80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- 172.21.15.105:0/1882510742 shutdown_connections 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- 172.21.15.105:0/1882510742 wait complete. 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 Processor -- start 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- start start 2024-04-22T06:37:40.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8101f80 0x7f8ba8198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8103b30 0x7f8ba819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba810db00 con 0x7f8ba8103170 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba810d980 con 0x7f8ba8103b30 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.716+0000 7f8bb0152640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba810dc80 con 0x7f8ba8101f80 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8badeee640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8101f80 0x7f8ba8198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34206/0 (socket says 172.21.15.105:34206) 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 -- 172.21.15.105:0/2648513973 learned_addr learned my addr 172.21.15.105:0/2648513973 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bae6ef640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8103b30 0x7f8ba819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 -- 172.21.15.105:0/2648513973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8101f80 msgr2=0x7f8ba8198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8101f80 0x7f8ba8198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 -- 172.21.15.105:0/2648513973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8103b30 msgr2=0x7f8ba819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8103b30 0x7f8ba819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 -- 172.21.15.105:0/2648513973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba819fda0 con 0x7f8ba8103170 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bad6ed640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800b4a0 tx=0x7f8b9800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98014070 con 0x7f8ba8103170 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b98010ac0 con 0x7f8ba8103170 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba81a0090 con 0x7f8ba8103170 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98017c30 con 0x7f8ba8103170 2024-04-22T06:37:40.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.717+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba81a05e0 con 0x7f8ba8103170 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.719+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba8109370 con 0x7f8ba8103170 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.719+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 138818+0+0 (secure 0 0 0) 0x7f8b98017070 con 0x7f8ba8103170 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.720+0000 7f8b9effd640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 0x7f8b940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.720+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f8b980cff10 con 0x7f8ba8103170 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.720+0000 7f8badeee640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 0x7f8b940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:40.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.720+0000 7f8badeee640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 0x7f8b940a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4005fd0 tx=0x7f8ba400b410 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:40.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.725+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b9809c5c0 con 0x7f8ba8103170 2024-04-22T06:37:40.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.959+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ba802d7b0 con 0x7f8ba8103170 2024-04-22T06:37:40.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.960+0000 7f8b9effd640 1 -- 172.21.15.105:0/2648513973 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f8b9809bec0 con 0x7f8ba8103170 2024-04-22T06:37:40.959 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:40.959 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:37:40.959 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:37:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 msgr2=0x7f8b940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 0x7f8b940a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4005fd0 tx=0x7f8ba400b410 comp rx=0 tx=0).stop 2024-04-22T06:37:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 msgr2=0x7f8ba81986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800b4a0 tx=0x7f8b9800b960 comp rx=0 tx=0).stop 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 shutdown_connections 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8b940a3db0 0x7f8b940a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8ba8103b30 0x7f8ba819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba8103170 0x7f8ba81986c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 --2- 172.21.15.105:0/2648513973 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ba8101f80 0x7f8ba8198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.963+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 >> 172.21.15.105:0/2648513973 conn(0x7f8ba80fd530 msgr2=0x7f8ba8106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.964+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 shutdown_connections 2024-04-22T06:37:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:40.964+0000 7f8bb0152640 1 -- 172.21.15.105:0/2648513973 wait complete. 2024-04-22T06:37:40.977 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-22T06:37:40.977 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-22T06:37:40.978 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:41.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 -- 172.21.15.105:0/561800402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 msgr2=0x7f02e8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 --2- 172.21.15.105:0/561800402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e8100c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009900 tx=0x7f02d802bcd0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 -- 172.21.15.105:0/561800402 shutdown_connections 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 --2- 172.21.15.105:0/561800402 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e8101b60 0x7f02e8101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 --2- 172.21.15.105:0/561800402 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e81011c0 0x7f02e8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 --2- 172.21.15.105:0/561800402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e8100c80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 -- 172.21.15.105:0/561800402 >> 172.21.15.105:0/561800402 conn(0x7f02e80fc3e0 msgr2=0x7f02e80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 -- 172.21.15.105:0/561800402 shutdown_connections 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.082+0000 7f02edf0e640 1 -- 172.21.15.105:0/561800402 wait complete. 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 Processor -- start 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 -- start start 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 0x7f02e819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 0x7f02e81a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f02e810fc80 con 0x7f02e8100860 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02e810fb00 con 0x7f02e8101b60 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02edf0e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02e810fe00 con 0x7f02e81011c0 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02e77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02e77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34212/0 (socket says 172.21.15.105:34212) 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02e77fe640 1 -- 172.21.15.105:0/4261643892 learned_addr learned my addr 172.21.15.105:0/4261643892 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.083+0000 7f02e7fff640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 0x7f02e81a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e6ffd640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 0x7f02e819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 -- 172.21.15.105:0/4261643892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 msgr2=0x7f02e819a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 0x7f02e819a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 -- 172.21.15.105:0/4261643892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 msgr2=0x7f02e81a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 0x7f02e81a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 -- 172.21.15.105:0/4261643892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e81a1fd0 con 0x7f02e8100860 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e6ffd640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 0x7f02e819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e7fff640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 0x7f02e81a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e77fe640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f02d8006010 tx=0x7f02d8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8043020 con 0x7f02e8100860 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02d802dc20 con 0x7f02e8100860 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e81a2260 con 0x7f02e8100860 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8003a10 con 0x7f02e8100860 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.084+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e81a27b0 con 0x7f02e8100860 2024-04-22T06:37:41.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.085+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e810b840 con 0x7f02e8100860 2024-04-22T06:37:41.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.086+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 138818+0+0 (secure 0 0 0) 0x7f02d8046020 con 0x7f02e8100860 2024-04-22T06:37:41.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.087+0000 7f02e4ff9640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 0x7f02d00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.087+0000 7f02e6ffd640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 0x7f02d00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.087+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f02d80f29f0 con 0x7f02e8100860 2024-04-22T06:37:41.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.087+0000 7f02e6ffd640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 0x7f02d00a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02e819b8c0 tx=0x7f02dc005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.091+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02d804d020 con 0x7f02e8100860 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1135122535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:37:41.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3452612331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2793154243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.211 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:40 smithi112 ceph-mon[31236]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1135122535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3452612331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2793154243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.242 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:40 smithi177 ceph-mon[31346]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.264 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.266+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f02d80bb010 con 0x7f02e8100860 2024-04-22T06:37:41.279 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.281+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f02e8062390 con 0x7f02d00a3db0 2024-04-22T06:37:41.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.289+0000 7f02e4ff9640 1 -- 172.21.15.105:0/4261643892 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257816 (secure 0 0 0) 0x7f02e8062390 con 0x7f02d00a3db0 2024-04-22T06:37:41.291 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:41.292 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:41.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 msgr2=0x7f02d00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 0x7f02d00a6270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02e819b8c0 tx=0x7f02dc005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 msgr2=0x7f02e819a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f02d8006010 tx=0x7f02d8005d50 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 shutdown_connections 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02d00a3db0 0x7f02d00a6270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02e8101b60 0x7f02e81a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02e81011c0 0x7f02e819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 --2- 172.21.15.105:0/4261643892 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02e8100860 0x7f02e819a3b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.291+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 >> 172.21.15.105:0/4261643892 conn(0x7f02e80fc3e0 msgr2=0x7f02e80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.292+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 shutdown_connections 2024-04-22T06:37:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.292+0000 7f02edf0e640 1 -- 172.21.15.105:0/4261643892 wait complete. 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1135122535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.301 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:41.302 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi105:/etc/ceph/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi112:/etc/ceph/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi177:/etc/ceph/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3452612331' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2793154243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:41.303 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.304 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.304 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:40 smithi105 ceph-mon[27916]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-04-22T06:37:41.304 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:37:40.942419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188971,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1388,"num_read_kb":858,"num_write":550,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":936,"ondisk_log_size":936,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385512,"kb_used_data":15524,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556312,"statfs":{"total":1151940427776,"available":1151545663488,"internally_reserved":0,"allocated":15896576,"data_stored":9311696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986572"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975254+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975254+0000","last_peered":"2024-04-22T06:37:36.975254+0000","last_clean":"2024-04-22T06:37:36.975254+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975254+0000","last_undegraded":"2024-04-22T06:37:36.975254+0000","last_fullsized":"2024-04-22T06:37:36.975254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.145529+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:37.145529+0000","last_peered":"2024-04-22T06:37:37.145529+0000","last_clean":"2024-04-22T06:37:37.145529+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:37.145529+0000","last_undegraded":"2024-04-22T06:37:37.145529+0000","last_fullsized":"2024-04-22T06:37:37.145529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974805+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974805+0000","last_peered":"2024-04-22T06:37:36.974805+0000","last_clean":"2024-04-22T06:37:36.974805+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974805+0000","last_undegraded":"2024-04-22T06:37:36.974805+0000","last_fullsized":"2024-04-22T06:37:36.974805+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146315+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:37.146315+0000","last_peered":"2024-04-22T06:37:37.146315+0000","last_clean":"2024-04-22T06:37:37.146315+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:37.146315+0000","last_undegraded":"2024-04-22T06:37:37.146315+0000","last_fullsized":"2024-04-22T06:37:37.146315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":251,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.169156+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:37.169156+0000","last_peered":"2024-04-22T06:37:37.169156+0000","last_clean":"2024-04-22T06:37:37.169156+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:37.169156+0000","last_undegraded":"2024-04-22T06:37:37.169156+0000","last_fullsized":"2024-04-22T06:37:37.169156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":291,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.181576+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:37.181576+0000","last_peered":"2024-04-22T06:37:37.181576+0000","last_clean":"2024-04-22T06:37:37.181576+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:37.181576+0000","last_undegraded":"2024-04-22T06:37:37.181576+0000","last_fullsized":"2024-04-22T06:37:37.181576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64727,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":234,"num_read_kb":181,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'9","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146596+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:37.146596+0000","last_peered":"2024-04-22T06:37:37.146596+0000","last_clean":"2024-04-22T06:37:37.146596+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:37.146596+0000","last_undegraded":"2024-04-22T06:37:37.146596+0000","last_fullsized":"2024-04-22T06:37:37.146596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1819,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933460+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933460+0000","last_peered":"2024-04-22T06:37:36.933460+0000","last_clean":"2024-04-22T06:37:36.933460+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933460+0000","last_undegraded":"2024-04-22T06:37:36.933460+0000","last_fullsized":"2024-04-22T06:37:36.933460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'6","reported_seq":139,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146001+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:37.146001+0000","last_peered":"2024-04-22T06:37:37.146001+0000","last_clean":"2024-04-22T06:37:37.146001+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:37.146001+0000","last_undegraded":"2024-04-22T06:37:37.146001+0000","last_fullsized":"2024-04-22T06:37:37.146001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975160+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:36.975160+0000","last_peered":"2024-04-22T06:37:36.975160+0000","last_clean":"2024-04-22T06:37:36.975160+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:36.975160+0000","last_undegraded":"2024-04-22T06:37:36.975160+0000","last_fullsized":"2024-04-22T06:37:36.975160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934327+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:36.934327+0000","last_peered":"2024-04-22T06:37:36.934327+0000","last_clean":"2024-04-22T06:37:36.934327+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:36.934327+0000","last_undegraded":"2024-04-22T06:37:36.934327+0000","last_fullsized":"2024-04-22T06:37:36.934327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975041+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975041+0000","last_peered":"2024-04-22T06:37:36.975041+0000","last_clean":"2024-04-22T06:37:36.975041+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975041+0000","last_undegraded":"2024-04-22T06:37:36.975041+0000","last_fullsized":"2024-04-22T06:37:36.975041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.132207+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.132207+0000","last_peered":"2024-04-22T06:37:37.132207+0000","last_clean":"2024-04-22T06:37:37.132207+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.132207+0000","last_undegraded":"2024-04-22T06:37:37.132207+0000","last_fullsized":"2024-04-22T06:37:37.132207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974018+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974018+0000","last_peered":"2024-04-22T06:37:36.974018+0000","last_clean":"2024-04-22T06:37:36.974018+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974018+0000","last_undegraded":"2024-04-22T06:37:36.974018+0000","last_fullsized":"2024-04-22T06:37:36.974018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146556+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:37.146556+0000","last_peered":"2024-04-22T06:37:37.146556+0000","last_clean":"2024-04-22T06:37:37.146556+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:37.146556+0000","last_undegraded":"2024-04-22T06:37:37.146556+0000","last_fullsized":"2024-04-22T06:37:37.146556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598369,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1252,"num_read_kb":739,"num_write":367,"num_write_kb":1460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1795107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252900,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514537,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37084,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708068,"statfs":{"total":95995035648,"available":95957061632,"internally_reserved":0,"allocated":1785856,"data_stored":1207741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710767,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972403,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1630208,"data_stored":1079932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":551162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":551705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1777664,"data_stored":1209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:41.309 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- 172.21.15.105:0/1901315781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc1011a0 msgr2=0x7f81cc101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1901315781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc1011a0 0x7f81cc101620 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009930 tx=0x7f81bc02bc20 comp rx=0 tx=0).stop 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- 172.21.15.105:0/1901315781 shutdown_connections 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1901315781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc101b60 0x7f81cc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1901315781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc1011a0 0x7f81cc101620 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1901315781 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- 172.21.15.105:0/1901315781 >> 172.21.15.105:0/1901315781 conn(0x7f81cc077e30 msgr2=0x7f81cc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- 172.21.15.105:0/1901315781 shutdown_connections 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- 172.21.15.105:0/1901315781 wait complete. 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 Processor -- start 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.454+0000 7f81d35f4640 1 -- start start 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 0x7f81cc1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.455 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc1110c0 con 0x7f81cc101b60 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc110f40 con 0x7f81cc1011a0 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc111240 con 0x7f81cc105e30 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34232/0 (socket says 172.21.15.105:34232) 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 -- 172.21.15.105:0/1687483194 learned_addr learned my addr 172.21.15.105:0/1687483194 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d1b91640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d1390640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 0x7f81cc1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 -- 172.21.15.105:0/1687483194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 msgr2=0x7f81cc19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 -- 172.21.15.105:0/1687483194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 msgr2=0x7f81cc1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 0x7f81cc1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 -- 172.21.15.105:0/1687483194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc19af70 con 0x7f81cc101b60 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d1390640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 0x7f81cc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:41.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d1b91640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d0b8f640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009230 tx=0x7f81bc004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc043020 con 0x7f81cc101b60 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81bc004350 con 0x7f81cc101b60 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc19b200 con 0x7f81cc101b60 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc003b50 con 0x7f81cc101b60 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.455+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc1a7570 con 0x7f81cc101b60 2024-04-22T06:37:41.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.456+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81cc19b570 con 0x7f81cc101b60 2024-04-22T06:37:41.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.458+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f81bc046020 con 0x7f81cc101b60 2024-04-22T06:37:41.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.458+0000 7f81c27fc640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 0x7f81b80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.458+0000 7f81d1390640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 0x7f81b80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.458+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f81bc0f2fa0 con 0x7f81cc101b60 2024-04-22T06:37:41.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.459+0000 7f81d1390640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 0x7f81b80a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81c800abe0 tx=0x7f81c800a5c0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.462+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81bc04d020 con 0x7f81cc101b60 2024-04-22T06:37:41.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.649+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81cc19bbe0 con 0x7f81b80a3e40 2024-04-22T06:37:41.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.657+0000 7f81c27fc640 1 -- 172.21.15.105:0/1687483194 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257816 (secure 0 0 0) 0x7f81cc19bbe0 con 0x7f81b80a3e40 2024-04-22T06:37:41.658 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:41.659 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 msgr2=0x7f81b80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 0x7f81b80a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81c800abe0 tx=0x7f81c800a5c0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 msgr2=0x7f81cc1a0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009230 tx=0x7f81bc004bb0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 shutdown_connections 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81b80a3e40 0x7f81b80a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81cc105e30 0x7f81cc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81cc101b60 0x7f81cc1a0b30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 --2- 172.21.15.105:0/1687483194 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81cc1011a0 0x7f81cc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 >> 172.21.15.105:0/1687483194 conn(0x7f81cc077e30 msgr2=0x7f81cc10ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:41.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 shutdown_connections 2024-04-22T06:37:41.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.660+0000 7f81d35f4640 1 -- 172.21.15.105:0/1687483194 wait complete. 2024-04-22T06:37:41.669 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:37:40.942419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188971,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1388,"num_read_kb":858,"num_write":550,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":936,"ondisk_log_size":936,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385512,"kb_used_data":15524,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556312,"statfs":{"total":1151940427776,"available":1151545663488,"internally_reserved":0,"allocated":15896576,"data_stored":9311696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986572"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975254+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975254+0000","last_peered":"2024-04-22T06:37:36.975254+0000","last_clean":"2024-04-22T06:37:36.975254+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975254+0000","last_undegraded":"2024-04-22T06:37:36.975254+0000","last_fullsized":"2024-04-22T06:37:36.975254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.145529+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:37.145529+0000","last_peered":"2024-04-22T06:37:37.145529+0000","last_clean":"2024-04-22T06:37:37.145529+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:37.145529+0000","last_undegraded":"2024-04-22T06:37:37.145529+0000","last_fullsized":"2024-04-22T06:37:37.145529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974805+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974805+0000","last_peered":"2024-04-22T06:37:36.974805+0000","last_clean":"2024-04-22T06:37:36.974805+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974805+0000","last_undegraded":"2024-04-22T06:37:36.974805+0000","last_fullsized":"2024-04-22T06:37:36.974805+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146315+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:37.146315+0000","last_peered":"2024-04-22T06:37:37.146315+0000","last_clean":"2024-04-22T06:37:37.146315+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:37.146315+0000","last_undegraded":"2024-04-22T06:37:37.146315+0000","last_fullsized":"2024-04-22T06:37:37.146315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":251,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.169156+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:37.169156+0000","last_peered":"2024-04-22T06:37:37.169156+0000","last_clean":"2024-04-22T06:37:37.169156+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:37.169156+0000","last_undegraded":"2024-04-22T06:37:37.169156+0000","last_fullsized":"2024-04-22T06:37:37.169156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":291,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.181576+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:37.181576+0000","last_peered":"2024-04-22T06:37:37.181576+0000","last_clean":"2024-04-22T06:37:37.181576+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:37.181576+0000","last_undegraded":"2024-04-22T06:37:37.181576+0000","last_fullsized":"2024-04-22T06:37:37.181576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64727,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":234,"num_read_kb":181,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'9","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146596+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:37.146596+0000","last_peered":"2024-04-22T06:37:37.146596+0000","last_clean":"2024-04-22T06:37:37.146596+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:37.146596+0000","last_undegraded":"2024-04-22T06:37:37.146596+0000","last_fullsized":"2024-04-22T06:37:37.146596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1819,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933460+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933460+0000","last_peered":"2024-04-22T06:37:36.933460+0000","last_clean":"2024-04-22T06:37:36.933460+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933460+0000","last_undegraded":"2024-04-22T06:37:36.933460+0000","last_fullsized":"2024-04-22T06:37:36.933460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'6","reported_seq":139,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146001+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:37.146001+0000","last_peered":"2024-04-22T06:37:37.146001+0000","last_clean":"2024-04-22T06:37:37.146001+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:37.146001+0000","last_undegraded":"2024-04-22T06:37:37.146001+0000","last_fullsized":"2024-04-22T06:37:37.146001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975160+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:36.975160+0000","last_peered":"2024-04-22T06:37:36.975160+0000","last_clean":"2024-04-22T06:37:36.975160+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:36.975160+0000","last_undegraded":"2024-04-22T06:37:36.975160+0000","last_fullsized":"2024-04-22T06:37:36.975160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934327+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:36.934327+0000","last_peered":"2024-04-22T06:37:36.934327+0000","last_clean":"2024-04-22T06:37:36.934327+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:36.934327+0000","last_undegraded":"2024-04-22T06:37:36.934327+0000","last_fullsized":"2024-04-22T06:37:36.934327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975041+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975041+0000","last_peered":"2024-04-22T06:37:36.975041+0000","last_clean":"2024-04-22T06:37:36.975041+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975041+0000","last_undegraded":"2024-04-22T06:37:36.975041+0000","last_fullsized":"2024-04-22T06:37:36.975041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.132207+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.132207+0000","last_peered":"2024-04-22T06:37:37.132207+0000","last_clean":"2024-04-22T06:37:37.132207+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.132207+0000","last_undegraded":"2024-04-22T06:37:37.132207+0000","last_fullsized":"2024-04-22T06:37:37.132207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974018+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974018+0000","last_peered":"2024-04-22T06:37:36.974018+0000","last_clean":"2024-04-22T06:37:36.974018+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974018+0000","last_undegraded":"2024-04-22T06:37:36.974018+0000","last_fullsized":"2024-04-22T06:37:36.974018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146556+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:37.146556+0000","last_peered":"2024-04-22T06:37:37.146556+0000","last_clean":"2024-04-22T06:37:37.146556+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:37.146556+0000","last_undegraded":"2024-04-22T06:37:37.146556+0000","last_fullsized":"2024-04-22T06:37:37.146556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598369,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1252,"num_read_kb":739,"num_write":367,"num_write_kb":1460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1795107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252900,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514537,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37084,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708068,"statfs":{"total":95995035648,"available":95957061632,"internally_reserved":0,"allocated":1785856,"data_stored":1207741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710767,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972403,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1630208,"data_stored":1079932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":551162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":551705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1777664,"data_stored":1209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:41.673 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-22T06:37:41.673 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.808+0000 7fd691dd0640 1 -- 172.21.15.105:0/1832953641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c069cb0 msgr2=0x7fd68c104900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.808+0000 7fd691dd0640 1 --2- 172.21.15.105:0/1832953641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c069cb0 0x7fd68c104900 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd684009ca0 tx=0x7fd68402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- 172.21.15.105:0/1832953641 shutdown_connections 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- 172.21.15.105:0/1832953641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c069cb0 0x7fd68c104900 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- 172.21.15.105:0/1832953641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- 172.21.15.105:0/1832953641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c106f10 0x7fd68c0690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- 172.21.15.105:0/1832953641 >> 172.21.15.105:0/1832953641 conn(0x7fd68c077e30 msgr2=0x7fd68c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- 172.21.15.105:0/1832953641 shutdown_connections 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- 172.21.15.105:0/1832953641 wait complete. 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 Processor -- start 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- start start 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c069cb0 0x7fd68c1a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c1187b0 con 0x7fd68c106f10 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c118630 con 0x7fd68c107380 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.809+0000 7fd691dd0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd68c118930 con 0x7fd68c069cb0 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34256/0 (socket says 172.21.15.105:34256) 2024-04-22T06:37:41.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 -- 172.21.15.105:0/3599002998 learned_addr learned my addr 172.21.15.105:0/3599002998 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68b7fe640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c069cb0 0x7fd68c1a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68bfff640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 -- 172.21.15.105:0/3599002998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c069cb0 msgr2=0x7fd68c1a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c069cb0 0x7fd68c1a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 -- 172.21.15.105:0/3599002998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 msgr2=0x7fd68c19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 -- 172.21.15.105:0/3599002998 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68c19f270 con 0x7fd68c106f10 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68bfff640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd68affd640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00e9f0 tx=0x7fd67c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00ccc0 con 0x7fd68c106f10 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67c0108e0 con 0x7fd68c106f10 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68c19f560 con 0x7fd68c106f10 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c018860 con 0x7fd68c106f10 2024-04-22T06:37:41.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.810+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68c1ab800 con 0x7fd68c106f10 2024-04-22T06:37:41.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.811+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68c068de0 con 0x7fd68c106f10 2024-04-22T06:37:41.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.812+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fd67c010430 con 0x7fd68c106f10 2024-04-22T06:37:41.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.813+0000 7fd688ff9640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 0x7fd6780a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:41.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.813+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fd67c0d0d90 con 0x7fd68c106f10 2024-04-22T06:37:41.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.813+0000 7fd68b7fe640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 0x7fd6780a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:41.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.813+0000 7fd68b7fe640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 0x7fd6780a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd67400bbe0 tx=0x7fd67400b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:41.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.818+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd67c09d250 con 0x7fd68c106f10 2024-04-22T06:37:41.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.992+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd68c0fed80 con 0x7fd6780a3e40 2024-04-22T06:37:41.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:41.999+0000 7fd688ff9640 1 -- 172.21.15.105:0/3599002998 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257816 (secure 0 0 0) 0x7fd68c0fed80 con 0x7fd6780a3e40 2024-04-22T06:37:42.000 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:42.001 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 msgr2=0x7fd6780a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 0x7fd6780a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd67400bbe0 tx=0x7fd67400b390 comp rx=0 tx=0).stop 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 msgr2=0x7fd68c1a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00e9f0 tx=0x7fd67c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 shutdown_connections 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6780a3e40 0x7fd6780a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd68c107380 0x7fd68c19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd68c106f10 0x7fd68c1a4e80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 --2- 172.21.15.105:0/3599002998 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd68c069cb0 0x7fd68c1a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 >> 172.21.15.105:0/3599002998 conn(0x7fd68c077e30 msgr2=0x7fd68c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 shutdown_connections 2024-04-22T06:37:42.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.002+0000 7fd691dd0640 1 -- 172.21.15.105:0/3599002998 wait complete. 2024-04-22T06:37:42.011 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-22T06:37:40.942419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188971,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1388,"num_read_kb":858,"num_write":550,"num_write_kb":4400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":936,"ondisk_log_size":936,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":385512,"kb_used_data":15524,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124556312,"statfs":{"total":1151940427776,"available":1151545663488,"internally_reserved":0,"allocated":15896576,"data_stored":9311696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.986572"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975254+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975254+0000","last_peered":"2024-04-22T06:37:36.975254+0000","last_clean":"2024-04-22T06:37:36.975254+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975254+0000","last_undegraded":"2024-04-22T06:37:36.975254+0000","last_fullsized":"2024-04-22T06:37:36.975254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.145529+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:37.145529+0000","last_peered":"2024-04-22T06:37:37.145529+0000","last_clean":"2024-04-22T06:37:37.145529+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:37.145529+0000","last_undegraded":"2024-04-22T06:37:37.145529+0000","last_fullsized":"2024-04-22T06:37:37.145529+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974805+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974805+0000","last_peered":"2024-04-22T06:37:36.974805+0000","last_clean":"2024-04-22T06:37:36.974805+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974805+0000","last_undegraded":"2024-04-22T06:37:36.974805+0000","last_fullsized":"2024-04-22T06:37:36.974805+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146315+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:37.146315+0000","last_peered":"2024-04-22T06:37:37.146315+0000","last_clean":"2024-04-22T06:37:37.146315+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:37.146315+0000","last_undegraded":"2024-04-22T06:37:37.146315+0000","last_fullsized":"2024-04-22T06:37:37.146315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":251,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.169156+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:37.169156+0000","last_peered":"2024-04-22T06:37:37.169156+0000","last_clean":"2024-04-22T06:37:37.169156+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:37.169156+0000","last_undegraded":"2024-04-22T06:37:37.169156+0000","last_fullsized":"2024-04-22T06:37:37.169156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":224,"num_read_kb":224,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":291,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.181576+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:37.181576+0000","last_peered":"2024-04-22T06:37:37.181576+0000","last_clean":"2024-04-22T06:37:37.181576+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:37.181576+0000","last_undegraded":"2024-04-22T06:37:37.181576+0000","last_fullsized":"2024-04-22T06:37:37.181576+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64727,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":234,"num_read_kb":181,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'9","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146596+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:37.146596+0000","last_peered":"2024-04-22T06:37:37.146596+0000","last_clean":"2024-04-22T06:37:37.146596+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:37.146596+0000","last_undegraded":"2024-04-22T06:37:37.146596+0000","last_fullsized":"2024-04-22T06:37:37.146596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1819,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933460+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933460+0000","last_peered":"2024-04-22T06:37:36.933460+0000","last_clean":"2024-04-22T06:37:36.933460+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933460+0000","last_undegraded":"2024-04-22T06:37:36.933460+0000","last_fullsized":"2024-04-22T06:37:36.933460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'6","reported_seq":139,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146001+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:37.146001+0000","last_peered":"2024-04-22T06:37:37.146001+0000","last_clean":"2024-04-22T06:37:37.146001+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:37.146001+0000","last_undegraded":"2024-04-22T06:37:37.146001+0000","last_fullsized":"2024-04-22T06:37:37.146001+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975160+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:36.975160+0000","last_peered":"2024-04-22T06:37:36.975160+0000","last_clean":"2024-04-22T06:37:36.975160+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:36.975160+0000","last_undegraded":"2024-04-22T06:37:36.975160+0000","last_fullsized":"2024-04-22T06:37:36.975160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934327+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:36.934327+0000","last_peered":"2024-04-22T06:37:36.934327+0000","last_clean":"2024-04-22T06:37:36.934327+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:36.934327+0000","last_undegraded":"2024-04-22T06:37:36.934327+0000","last_fullsized":"2024-04-22T06:37:36.934327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975041+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975041+0000","last_peered":"2024-04-22T06:37:36.975041+0000","last_clean":"2024-04-22T06:37:36.975041+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975041+0000","last_undegraded":"2024-04-22T06:37:36.975041+0000","last_fullsized":"2024-04-22T06:37:36.975041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.132207+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.132207+0000","last_peered":"2024-04-22T06:37:37.132207+0000","last_clean":"2024-04-22T06:37:37.132207+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.132207+0000","last_undegraded":"2024-04-22T06:37:37.132207+0000","last_fullsized":"2024-04-22T06:37:37.132207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974018+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974018+0000","last_peered":"2024-04-22T06:37:36.974018+0000","last_clean":"2024-04-22T06:37:36.974018+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974018+0000","last_undegraded":"2024-04-22T06:37:36.974018+0000","last_fullsized":"2024-04-22T06:37:36.974018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":141,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.146556+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:37.146556+0000","last_peered":"2024-04-22T06:37:37.146556+0000","last_clean":"2024-04-22T06:37:37.146556+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:37.146556+0000","last_undegraded":"2024-04-22T06:37:37.146556+0000","last_fullsized":"2024-04-22T06:37:37.146556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598369,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1252,"num_read_kb":739,"num_write":367,"num_write_kb":1460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2162688,"data_stored":1795107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252900,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514537,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37084,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708068,"statfs":{"total":95995035648,"available":95957061632,"internally_reserved":0,"allocated":1785856,"data_stored":1207741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710767,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972403,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1630208,"data_stored":1079932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":551162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":551705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32912,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712240,"statfs":{"total":95995035648,"available":95961333760,"internally_reserved":0,"allocated":1777664,"data_stored":1209588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":199294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:42.016 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:42.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 -- 172.21.15.105:0/250159163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4406a8d0 msgr2=0x7f7d4406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 --2- 172.21.15.105:0/250159163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4406a8d0 0x7f7d4406acb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009900 tx=0x7f7d2c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:42.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 -- 172.21.15.105:0/250159163 shutdown_connections 2024-04-22T06:37:42.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 --2- 172.21.15.105:0/250159163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d441072b0 0x7f7d4410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 --2- 172.21.15.105:0/250159163 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d4406b280 0x7f7d44106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 --2- 172.21.15.105:0/250159163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4406a8d0 0x7f7d4406acb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 -- 172.21.15.105:0/250159163 >> 172.21.15.105:0/250159163 conn(0x7f7d44077c20 msgr2=0x7f7d44078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 -- 172.21.15.105:0/250159163 shutdown_connections 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 -- 172.21.15.105:0/250159163 wait complete. 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.153+0000 7f7d49625640 1 Processor -- start 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 -- start start 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 0x7f7d44198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d441072b0 0x7f7d44198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7d441121c0 con 0x7f7d4419d020 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d44112040 con 0x7f7d4406a8d0 2024-04-22T06:37:42.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d49625640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7d44112340 con 0x7f7d441072b0 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34284/0 (socket says 172.21.15.105:34284) 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 -- 172.21.15.105:0/864580385 learned_addr learned my addr 172.21.15.105:0/864580385 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d427fc640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d441072b0 0x7f7d44198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d42ffd640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 0x7f7d44198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 -- 172.21.15.105:0/864580385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d441072b0 msgr2=0x7f7d44198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d441072b0 0x7f7d44198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 -- 172.21.15.105:0/864580385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 msgr2=0x7f7d44198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 0x7f7d44198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 -- 172.21.15.105:0/864580385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4419db10 con 0x7f7d4419d020 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d42ffd640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 0x7f7d44198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d437fe640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00eb10 tx=0x7f7d3c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.154+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c004b20 con 0x7f7d4419d020 2024-04-22T06:37:42.155 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.155+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4419de00 con 0x7f7d4419d020 2024-04-22T06:37:42.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.155+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d3c004cc0 con 0x7f7d4419d020 2024-04-22T06:37:42.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.155+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c010670 con 0x7f7d4419d020 2024-04-22T06:37:42.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.155+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d441a55a0 con 0x7f7d4419d020 2024-04-22T06:37:42.156 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.156+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d44069230 con 0x7f7d4419d020 2024-04-22T06:37:42.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.157+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f7d3c005000 con 0x7f7d4419d020 2024-04-22T06:37:42.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.157+0000 7f7d3bfff640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 0x7f7d300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.157+0000 7f7d42ffd640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 0x7f7d300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.157+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f7d3c0d1210 con 0x7f7d4419d020 2024-04-22T06:37:42.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.158+0000 7f7d42ffd640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 0x7f7d300a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005ab0 tx=0x7f7d2c005a20 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.161+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d3c053ff0 con 0x7f7d4419d020 2024-04-22T06:37:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:42.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2648513973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:41 smithi112 ceph-mon[31236]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2648513973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:41 smithi177 ceph-mon[31346]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.337+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d44199ab0 con 0x7f7d4419d020 2024-04-22T06:37:42.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.339+0000 7f7d3bfff640 1 -- 172.21.15.105:0/864580385 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7f7d44199ab0 con 0x7f7d4419d020 2024-04-22T06:37:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:42.338 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 msgr2=0x7f7d300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 0x7f7d300a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005ab0 tx=0x7f7d2c005a20 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 msgr2=0x7f7d4419d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00eb10 tx=0x7f7d3c00c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 shutdown_connections 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7d300a3e40 0x7f7d300a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d4419d020 0x7f7d4419d4d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d441072b0 0x7f7d44198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 --2- 172.21.15.105:0/864580385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d4406a8d0 0x7f7d44198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 >> 172.21.15.105:0/864580385 conn(0x7f7d44077c20 msgr2=0x7f7d440ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 shutdown_connections 2024-04-22T06:37:42.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.341+0000 7f7d49625640 1 -- 172.21.15.105:0/864580385 wait complete. 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2648513973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: Updating smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: Updating smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: Updating smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.350 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:37:42.351 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:41 smithi105 ceph-mon[27916]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:42.353 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:42.353 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 -- 172.21.15.105:0/2625824239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100450 msgr2=0x7f35bc1008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 --2- 172.21.15.105:0/2625824239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100450 0x7f35bc1008d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009930 tx=0x7f35b002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 -- 172.21.15.105:0/2625824239 shutdown_connections 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 --2- 172.21.15.105:0/2625824239 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100e10 0x7f35bc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 --2- 172.21.15.105:0/2625824239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100450 0x7f35bc1008d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 --2- 172.21.15.105:0/2625824239 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 0x7f35bc107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 -- 172.21.15.105:0/2625824239 >> 172.21.15.105:0/2625824239 conn(0x7f35bc077410 msgr2=0x7f35bc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 -- 172.21.15.105:0/2625824239 shutdown_connections 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.455+0000 7f35c198f640 1 -- 172.21.15.105:0/2625824239 wait complete. 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 Processor -- start 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 -- start start 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 0x7f35bc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 0x7f35bc198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1122a0 con 0x7f35bc100e10 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc112120 con 0x7f35bc106f30 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35c198f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc112420 con 0x7f35bc100450 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35ba7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35ba7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34304/0 (socket says 172.21.15.105:34304) 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35ba7fc640 1 -- 172.21.15.105:0/3491620068 learned_addr learned my addr 172.21.15.105:0/3491620068 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.456+0000 7f35bb7fe640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 0x7f35bc198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35baffd640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 0x7f35bc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 -- 172.21.15.105:0/3491620068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 msgr2=0x7f35bc19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 0x7f35bc19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 -- 172.21.15.105:0/3491620068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 msgr2=0x7f35bc198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 0x7f35bc198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 -- 172.21.15.105:0/3491620068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35bc198f00 con 0x7f35bc100e10 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35baffd640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 0x7f35bc19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35ba7fc640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009230 tx=0x7f35b002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0043020 con 0x7f35bc100e10 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35b002caf0 con 0x7f35bc100e10 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35bc199190 con 0x7f35bc100e10 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0003af0 con 0x7f35bc100e10 2024-04-22T06:37:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.457+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35bc1a5500 con 0x7f35bc100e10 2024-04-22T06:37:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.458+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35bc103770 con 0x7f35bc100e10 2024-04-22T06:37:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.459+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f35b0046020 con 0x7f35bc100e10 2024-04-22T06:37:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.460+0000 7f35c098d640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 0x7f35a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.460+0000 7f35baffd640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 0x7f35a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.460+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f35b00f2ee0 con 0x7f35bc100e10 2024-04-22T06:37:42.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.460+0000 7f35baffd640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 0x7f35a40a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009b50 tx=0x7f35ac005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.464+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35b004d020 con 0x7f35bc100e10 2024-04-22T06:37:42.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.641+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35bc199b70 con 0x7f35bc100e10 2024-04-22T06:37:42.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.641+0000 7f35c098d640 1 -- 172.21.15.105:0/3491620068 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f35b00bf4b0 con 0x7f35bc100e10 2024-04-22T06:37:42.640 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 msgr2=0x7f35a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 0x7f35a40a6300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009b50 tx=0x7f35ac005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 msgr2=0x7f35bc19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009230 tx=0x7f35b002c710 comp rx=0 tx=0).stop 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 shutdown_connections 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35a40a3e40 0x7f35a40a6300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35bc106f30 0x7f35bc198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35bc100e10 0x7f35bc19eac0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 --2- 172.21.15.105:0/3491620068 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35bc100450 0x7f35bc19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 >> 172.21.15.105:0/3491620068 conn(0x7f35bc077410 msgr2=0x7f35bc105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 shutdown_connections 2024-04-22T06:37:42.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.643+0000 7f35c198f640 1 -- 172.21.15.105:0/3491620068 wait complete. 2024-04-22T06:37:42.652 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.754+0000 7f645ee81640 1 -- 172.21.15.105:0/2407889731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458102180 msgr2=0x7f6458102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.754+0000 7f645ee81640 1 --2- 172.21.15.105:0/2407889731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458102180 0x7f6458102560 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6448009900 tx=0x7f644802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- 172.21.15.105:0/2407889731 shutdown_connections 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- 172.21.15.105:0/2407889731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6458107250 0x7f645810ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- 172.21.15.105:0/2407889731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6458102b30 0x7f6458106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- 172.21.15.105:0/2407889731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458102180 0x7f6458102560 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- 172.21.15.105:0/2407889731 >> 172.21.15.105:0/2407889731 conn(0x7f64580fd9c0 msgr2=0x7f64580ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- 172.21.15.105:0/2407889731 shutdown_connections 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- 172.21.15.105:0/2407889731 wait complete. 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 Processor -- start 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- start start 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6458102180 0x7f64581a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f645819a7b0 0x7f645819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6458112190 con 0x7f6458107250 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6458112010 con 0x7f6458102180 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f645ee81640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6458112310 con 0x7f645819a7b0 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f6457fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f6457fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34320/0 (socket says 172.21.15.105:34320) 2024-04-22T06:37:42.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.755+0000 7f6457fff640 1 -- 172.21.15.105:0/3582619173 learned_addr learned my addr 172.21.15.105:0/3582619173 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f645d41e640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f645819a7b0 0x7f645819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 -- 172.21.15.105:0/3582619173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f645819a7b0 msgr2=0x7f645819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f645819a7b0 0x7f645819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 -- 172.21.15.105:0/3582619173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6458102180 msgr2=0x7f64581a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6458102180 0x7f64581a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 -- 172.21.15.105:0/3582619173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645819b340 con 0x7f6458107250 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6457fff640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c970 tx=0x7f644c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c007bb0 con 0x7f6458107250 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f644c007d50 con 0x7f6458107250 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c00f450 con 0x7f6458107250 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645819b630 con 0x7f6458107250 2024-04-22T06:37:42.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.756+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64581a7860 con 0x7f6458107250 2024-04-22T06:37:42.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.757+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6458103650 con 0x7f6458107250 2024-04-22T06:37:42.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.758+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f644c004f20 con 0x7f6458107250 2024-04-22T06:37:42.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.759+0000 7f6455ffb640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 0x7f64400a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:42.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.759+0000 7f645cc1d640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 0x7f64400a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:42.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.759+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f644c0d09f0 con 0x7f6458107250 2024-04-22T06:37:42.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.759+0000 7f645cc1d640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 0x7f64400a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6448005770 tx=0x7f6448000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:42.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.763+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f645819b630 con 0x7f6458107250 2024-04-22T06:37:42.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.940+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6458102b30 con 0x7f6458107250 2024-04-22T06:37:42.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.940+0000 7f6455ffb640 1 -- 172.21.15.105:0/3582619173 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f644c014030 con 0x7f6458107250 2024-04-22T06:37:42.939 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 msgr2=0x7f64400a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 0x7f64400a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6448005770 tx=0x7f6448000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 msgr2=0x7f64581a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f644c00c970 tx=0x7f644c00ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 shutdown_connections 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f64400a3ec0 0x7f64400a6380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f645819a7b0 0x7f645819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6458107250 0x7f64581a0b80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 --2- 172.21.15.105:0/3582619173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6458102180 0x7f64581a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:42.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 >> 172.21.15.105:0/3582619173 conn(0x7f64580fd9c0 msgr2=0x7f64580feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:42.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 shutdown_connections 2024-04-22T06:37:42.942 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:42.942+0000 7f645ee81640 1 -- 172.21.15.105:0/3582619173 wait complete. 2024-04-22T06:37:42.953 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:43.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.055+0000 7fd6d77b6640 1 -- 172.21.15.105:0/3725400806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01072b0 msgr2=0x7fd6d010ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.055+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/3725400806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01072b0 0x7fd6d010ee80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009ca0 tx=0x7fd6c802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- 172.21.15.105:0/3725400806 shutdown_connections 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/3725400806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01072b0 0x7fd6d010ee80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/3725400806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0102b90 0x7fd6d0106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/3725400806 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01021e0 0x7fd6d01025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- 172.21.15.105:0/3725400806 >> 172.21.15.105:0/3725400806 conn(0x7fd6d00fda60 msgr2=0x7fd6d00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- 172.21.15.105:0/3725400806 shutdown_connections 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- 172.21.15.105:0/3725400806 wait complete. 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 Processor -- start 2024-04-22T06:37:43.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- start start 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01072b0 0x7fd6d019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 0x7fd6d0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0112220 con 0x7fd6d01021e0 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01120a0 con 0x7fd6d0198600 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.056+0000 7fd6d77b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d01123a0 con 0x7fd6d01072b0 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34328/0 (socket says 172.21.15.105:34328) 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 -- 172.21.15.105:0/2172411492 learned_addr learned my addr 172.21.15.105:0/2172411492 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d4d51640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01072b0 0x7fd6d019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5d53640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 0x7fd6d0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 -- 172.21.15.105:0/2172411492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01072b0 msgr2=0x7fd6d019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01072b0 0x7fd6d019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 -- 172.21.15.105:0/2172411492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 msgr2=0x7fd6d0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 0x7fd6d0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 -- 172.21.15.105:0/2172411492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d0199120 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5d53640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 0x7fd6d0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d5552640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c970 tx=0x7fd6cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc007bb0 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6cc007d50 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d0199410 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00f450 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.057+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d01a55c0 con 0x7fd6d01021e0 2024-04-22T06:37:43.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.058+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d00691c0 con 0x7fd6d01021e0 2024-04-22T06:37:43.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.059+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fd6cc004f20 con 0x7fd6d01021e0 2024-04-22T06:37:43.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.060+0000 7fd6c67fc640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 0x7fd6bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.060+0000 7fd6d4d51640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 0x7fd6bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.060+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fd6cc0d07c0 con 0x7fd6d01021e0 2024-04-22T06:37:43.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.060+0000 7fd6d4d51640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 0x7fd6bc0a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6d01011b0 tx=0x7fd6c0005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.064+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6cc014030 con 0x7fd6d01021e0 2024-04-22T06:37:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:42 smithi112 ceph-mon[31236]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/864580385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3491620068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:42 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3582619173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.238+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6d0102b90 con 0x7fd6d01021e0 2024-04-22T06:37:43.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.239+0000 7fd6c67fc640 1 -- 172.21.15.105:0/2172411492 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fd6cc09cc80 con 0x7fd6d01021e0 2024-04-22T06:37:43.237 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 msgr2=0x7fd6bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 0x7fd6bc0a6300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd6d01011b0 tx=0x7fd6c0005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 msgr2=0x7fd6d019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c970 tx=0x7fd6cc00ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 shutdown_connections 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd6bc0a3e40 0x7fd6bc0a6300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd6d0198600 0x7fd6d0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d01072b0 0x7fd6d019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 --2- 172.21.15.105:0/2172411492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6d01021e0 0x7fd6d019e400 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 >> 172.21.15.105:0/2172411492 conn(0x7fd6d00fda60 msgr2=0x7fd6d00ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 shutdown_connections 2024-04-22T06:37:43.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.241+0000 7fd6d77b6640 1 -- 172.21.15.105:0/2172411492 wait complete. 2024-04-22T06:37:43.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:42 smithi177 ceph-mon[31346]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:43.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/864580385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:43.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3491620068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:42 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3582619173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.249 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:42 smithi105 ceph-mon[27916]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:43.249 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/864580385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:43.249 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3491620068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.249 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:42 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3582619173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:43.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.350+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/480140106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c8069aa0 msgr2=0x7fc4c810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.350+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/480140106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c8069aa0 0x7fc4c810ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009ca0 tx=0x7fc4c002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/480140106 shutdown_connections 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/480140106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c8069aa0 0x7fc4c810ede0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/480140106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4c80690e0 0x7fc4c8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/480140106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4c8106f50 0x7fc4c8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/480140106 >> 172.21.15.105:0/480140106 conn(0x7fc4c8077e30 msgr2=0x7fc4c8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/480140106 shutdown_connections 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/480140106 wait complete. 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 Processor -- start 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- start start 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4c8069aa0 0x7fc4c81a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4c8106f50 0x7fc4c819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c81121d0 con 0x7fc4c80690e0 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c8112050 con 0x7fc4c8106f50 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cf9f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c8112350 con 0x7fc4c8069aa0 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cd790640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cd790640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34358/0 (socket says 172.21.15.105:34358) 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.351+0000 7fc4cd790640 1 -- 172.21.15.105:0/2394520362 learned_addr learned my addr 172.21.15.105:0/2394520362 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:43.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 -- 172.21.15.105:0/2394520362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4c8069aa0 msgr2=0x7fc4c81a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4c8069aa0 0x7fc4c81a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 -- 172.21.15.105:0/2394520362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4c8106f50 msgr2=0x7fc4c819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4c8106f50 0x7fc4c819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 -- 172.21.15.105:0/2394520362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c819af30 con 0x7fc4c80690e0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cd790640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800d950 tx=0x7fc4b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8017070 con 0x7fc4c80690e0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4b8009c30 con 0x7fc4c80690e0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c819b220 con 0x7fc4c80690e0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800f620 con 0x7fc4c80690e0 2024-04-22T06:37:43.353 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.352+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c81a7530 con 0x7fc4c80690e0 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.353+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c8103650 con 0x7fc4c80690e0 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.354+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fc4b8010460 con 0x7fc4c80690e0 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.355+0000 7fc4c67fc640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 0x7fc4b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.355+0000 7fc4ccf8f640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 0x7fc4b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.355+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc4b80d0750 con 0x7fc4c80690e0 2024-04-22T06:37:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.355+0000 7fc4ccf8f640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 0x7fc4b00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c819bec0 tx=0x7fc4bc006c90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.357 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.359+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4b809cc10 con 0x7fc4c80690e0 2024-04-22T06:37:43.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.572+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4c8100270 con 0x7fc4c80690e0 2024-04-22T06:37:43.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.573+0000 7fc4c67fc640 1 -- 172.21.15.105:0/2394520362 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fc4b8009de0 con 0x7fc4c80690e0 2024-04-22T06:37:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:43.572 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:37:43.572 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 msgr2=0x7fc4b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 0x7fc4b00a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4c819bec0 tx=0x7fc4bc006c90 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 msgr2=0x7fc4c81a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800d950 tx=0x7fc4b800de10 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 shutdown_connections 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4b00a3e40 0x7fc4b00a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4c8106f50 0x7fc4c819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4c8069aa0 0x7fc4c81a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 --2- 172.21.15.105:0/2394520362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4c80690e0 0x7fc4c81a05b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 >> 172.21.15.105:0/2394520362 conn(0x7fc4c8077e30 msgr2=0x7fc4c80fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 shutdown_connections 2024-04-22T06:37:43.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.575+0000 7fc4cf9f4640 1 -- 172.21.15.105:0/2394520362 wait complete. 2024-04-22T06:37:43.585 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 -- 172.21.15.105:0/3611879628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64069a80 msgr2=0x7f3c6410ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/3611879628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64069a80 0x7f3c6410ed80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009cd0 tx=0x7f3c5c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 -- 172.21.15.105:0/3611879628 shutdown_connections 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/3611879628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64069a80 0x7f3c6410ed80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/3611879628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 0x7f3c64069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/3611879628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64107f70 0x7f3c64108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 -- 172.21.15.105:0/3611879628 >> 172.21.15.105:0/3611879628 conn(0x7f3c64077e30 msgr2=0x7f3c64078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 -- 172.21.15.105:0/3611879628 shutdown_connections 2024-04-22T06:37:43.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.684+0000 7f3c68e7b640 1 -- 172.21.15.105:0/3611879628 wait complete. 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 Processor -- start 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 -- start start 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 0x7f3c64101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64069a80 0x7f3c641017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64112150 con 0x7f3c64107f70 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c64111fd0 con 0x7f3c64069a80 2024-04-22T06:37:43.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c68e7b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c641122d0 con 0x7f3c640690e0 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.685+0000 7f3c62d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34372/0 (socket says 172.21.15.105:34372) 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 -- 172.21.15.105:0/2146998714 learned_addr learned my addr 172.21.15.105:0/2146998714 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c6259c640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 0x7f3c64101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c59d9b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64069a80 0x7f3c641017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 -- 172.21.15.105:0/2146998714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 msgr2=0x7f3c64101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 0x7f3c64101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 -- 172.21.15.105:0/2146998714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64069a80 msgr2=0x7f3c641017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64069a80 0x7f3c641017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 -- 172.21.15.105:0/2146998714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c641a6b10 con 0x7f3c64107f70 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c62d9d640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c033000 tx=0x7f3c5c0346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c041020 con 0x7f3c64107f70 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c5c032970 con 0x7f3c64107f70 2024-04-22T06:37:43.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c641a6da0 con 0x7f3c64107f70 2024-04-22T06:37:43.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c039040 con 0x7f3c64107f70 2024-04-22T06:37:43.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.686+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c641a7290 con 0x7f3c64107f70 2024-04-22T06:37:43.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.687+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c64068de0 con 0x7f3c64107f70 2024-04-22T06:37:43.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.688+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f3c5c044020 con 0x7f3c64107f70 2024-04-22T06:37:43.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.689+0000 7f3c5bfff640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:43.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.689+0000 7f3c6259c640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:43.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.689+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f3c5c0f26e0 con 0x7f3c64107f70 2024-04-22T06:37:43.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.689+0000 7f3c6259c640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 0x7f3c500a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005d00 tx=0x7f3c5400b410 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:43.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.693+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c5c04b020 con 0x7f3c64107f70 2024-04-22T06:37:43.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.872+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c6410a3d0 con 0x7f3c64107f70 2024-04-22T06:37:43.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.874+0000 7f3c5bfff640 1 -- 172.21.15.105:0/2146998714 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7f3c5c0becb0 con 0x7f3c64107f70 2024-04-22T06:37:43.872 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:43.872 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:43.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 msgr2=0x7f3c500a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 0x7f3c500a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005d00 tx=0x7f3c5400b410 comp rx=0 tx=0).stop 2024-04-22T06:37:43.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 msgr2=0x7f3c64101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:43.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c033000 tx=0x7f3c5c0346e0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 shutdown_connections 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c500a3e40 0x7f3c500a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c64107f70 0x7f3c64101cf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c64069a80 0x7f3c641017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 --2- 172.21.15.105:0/2146998714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c640690e0 0x7f3c64101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 >> 172.21.15.105:0/2146998714 conn(0x7f3c64077e30 msgr2=0x7f3c640fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 shutdown_connections 2024-04-22T06:37:43.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:43.876+0000 7f3c68e7b640 1 -- 172.21.15.105:0/2146998714 wait complete. 2024-04-22T06:37:43.886 DEBUG:tasks.fwd_scrub:Yielding 2024-04-22T06:37:43.886 INFO:teuthology.run_tasks:Running task exec... 2024-04-22T06:37:43.899 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:37:43.899 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:43.900 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-22T06:37:43.901 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-22T06:37:43.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:37:43.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:37:43.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:37:44.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.005+0000 7fd885fa4640 1 -- 172.21.15.105:0/98956355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 msgr2=0x7fd8801035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.005+0000 7fd885fa4640 1 --2- 172.21.15.105:0/98956355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd8801035c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd870009ca0 tx=0x7fd87002c230 comp rx=0 tx=0).stop 2024-04-22T06:37:44.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 -- 172.21.15.105:0/98956355 shutdown_connections 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 --2- 172.21.15.105:0/98956355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd88010c520 0x7fd88010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 --2- 172.21.15.105:0/98956355 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd880103b00 0x7fd88010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 --2- 172.21.15.105:0/98956355 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd8801035c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 -- 172.21.15.105:0/98956355 >> 172.21.15.105:0/98956355 conn(0x7fd880077fd0 msgr2=0x7fd8800783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 -- 172.21.15.105:0/98956355 shutdown_connections 2024-04-22T06:37:44.005 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 -- 172.21.15.105:0/98956355 wait complete. 2024-04-22T06:37:44.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.006+0000 7fd885fa4640 1 Processor -- start 2024-04-22T06:37:44.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 -- start start 2024-04-22T06:37:44.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd880103b00 0x7fd880198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 0x7fd88019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd8801121b0 con 0x7fd8801031e0 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112030 con 0x7fd88010c520 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd885fa4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd880112330 con 0x7fd880103b00 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34402/0 (socket says 172.21.15.105:34402) 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 -- 172.21.15.105:0/4191359411 learned_addr learned my addr 172.21.15.105:0/4191359411 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87effd640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd880103b00 0x7fd880198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87ffff640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 0x7fd88019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 -- 172.21.15.105:0/4191359411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd880103b00 msgr2=0x7fd880198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd880103b00 0x7fd880198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 -- 172.21.15.105:0/4191359411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 msgr2=0x7fd88019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 0x7fd88019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 -- 172.21.15.105:0/4191359411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88019d6b0 con 0x7fd8801031e0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87ffff640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 0x7fd88019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.007+0000 7fd87f7fe640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd870005e30 tx=0x7fd870003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd870039040 con 0x7fd8801031e0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd870044070 con 0x7fd8801031e0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88019d940 con 0x7fd8801031e0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8700328a0 con 0x7fd8801031e0 2024-04-22T06:37:44.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8801a5230 con 0x7fd8801031e0 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 -- 172.21.15.105:0/1183821090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c44104db0 msgr2=0x7f3c44108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 --2- 172.21.15.105:0/1183821090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c44104db0 0x7f3c44108df0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009900 tx=0x7f3c2c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 -- 172.21.15.105:0/1183821090 shutdown_connections 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 --2- 172.21.15.105:0/1183821090 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c441094d0 0x7f3c441110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 --2- 172.21.15.105:0/1183821090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c44104db0 0x7f3c44108df0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 --2- 172.21.15.105:0/1183821090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 0x7f3c441047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 -- 172.21.15.105:0/1183821090 >> 172.21.15.105:0/1183821090 conn(0x7f3c440ffce0 msgr2=0x7f3c44102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 -- 172.21.15.105:0/1183821090 shutdown_connections 2024-04-22T06:37:44.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.008+0000 7f3c490e0640 1 -- 172.21.15.105:0/1183821090 wait complete. 2024-04-22T06:37:44.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.009+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd880069e00 con 0x7fd8801031e0 2024-04-22T06:37:44.012 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.009+0000 7f3c490e0640 1 Processor -- start 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.009+0000 7f3c490e0640 1 -- start start 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 0x7f3c441a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 0x7f3c4419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441143f0 con 0x7f3c441094d0 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c44114270 con 0x7f3c44104400 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c490e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c44114570 con 0x7f3c4419a8b0 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34418/0 (socket says 172.21.15.105:34418) 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3333433475 learned_addr learned my addr 172.21.15.105:0/3333433475 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 0x7f3c441a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c4359e640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 0x7f3c4419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3333433475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 msgr2=0x7f3c4419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 0x7f3c4419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3333433475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 msgr2=0x7f3c441a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 0x7f3c441a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3333433475 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4419b3d0 con 0x7f3c441094d0 2024-04-22T06:37:44.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c4359e640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 0x7f3c4419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:44.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c036040 tx=0x7f3c2c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.010+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fd870004240 con 0x7fd8801031e0 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.011+0000 7fd87cff9640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 0x7fd86c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.011+0000 7fd87effd640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 0x7fd86c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.011+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fd8700f3a70 con 0x7fd8801031e0 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.011+0000 7fd87effd640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 0x7fd86c0a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd8800fff10 tx=0x7fd86800a2d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7f3c40d99640 1 -- 172.21.15.105:0/3333433475 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c043020 con 0x7f3c441094d0 2024-04-22T06:37:44.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7f3c40d99640 1 -- 172.21.15.105:0/3333433475 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c2c02cbe0 con 0x7f3c441094d0 2024-04-22T06:37:44.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7f3c40d99640 1 -- 172.21.15.105:0/3333433475 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c003bf0 con 0x7f3c441094d0 2024-04-22T06:37:44.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4419b660 con 0x7f3c441094d0 2024-04-22T06:37:44.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8700c0160 con 0x7fd8801031e0 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.014+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441a78b0 con 0x7f3c441094d0 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 Processor -- start 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 -- start start 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a20840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a23260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c40d99640 1 -- 172.21.15.105:0/3333433475 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f3c2c046020 con 0x7f3c441094d0 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556e9497bf00 con 0x556e94a178e0 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556e94919590 con 0x556e94a20e10 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.017+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e94977ec0 con 0x556e94a18290 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34428/0 (socket says 172.21.15.105:34428) 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 -- 172.21.15.105:0/678486 learned_addr learned my addr 172.21.15.105:0/678486 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a20840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a23260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 -- 172.21.15.105:0/678486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 msgr2=0x556e94a20840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a20840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 -- 172.21.15.105:0/678486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 msgr2=0x556e94a23260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a23260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 -- 172.21.15.105:0/678486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e947567e0 con 0x556e94a178e0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a23260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c4359e640 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009f30 tx=0x7f3c3c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a632d7d60f4c8af server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c40d99640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 0x7f3c300a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c397fa640 1 -- 172.21.15.105:0/678486 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c016070 con 0x556e94a178e0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 0x7f3c300a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c397fa640 1 -- 172.21.15.105:0/678486 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c3c01bb10 con 0x556e94a178e0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c397fa640 1 -- 172.21.15.105:0/678486 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c012bc0 con 0x556e94a178e0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.018+0000 7f3c40d99640 1 -- 172.21.15.105:0/3333433475 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f3c2c0f2b20 con 0x7f3c441094d0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/678486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 msgr2=0x556e94a17cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009f30 tx=0x7f3c3c010800 comp rx=0 tx=0).stop 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 0x7f3c300a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x556e94a13020 tx=0x7f3c3400a2d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/678486 shutdown_connections 2024-04-22T06:37:44.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a23260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a20840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/678486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a17cc0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/678486 >> 172.21.15.105:0/678486 conn(0x556e94a11fb0 msgr2=0x556e94a12390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/678486 shutdown_connections 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.019+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/678486 wait complete. 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 -- start start 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 Processor -- start 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a178e0 0x556e94a1de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a18290 0x556e94a1e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556e94a23d80 con 0x556e94a20e10 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556e94935580 con 0x556e94a18290 2024-04-22T06:37:44.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556e94a23f00 con 0x556e94a178e0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4bdaf740 10 client.0 _ll_register_callbacks cb 0x556e9493b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34434/0 (socket says 172.21.15.105:34434) 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 -- 172.21.15.105:0/680618530 learned_addr learned my addr 172.21.15.105:0/680618530 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a178e0 0x556e94a1de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a18290 0x556e94a1e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 -- 172.21.15.105:0/680618530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a178e0 msgr2=0x556e94a1de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a178e0 0x556e94a1de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 -- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a18290 msgr2=0x556e94a1e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a18290 0x556e94a1e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.020+0000 7f3c4359e640 1 -- 172.21.15.105:0/680618530 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e947b6f70 con 0x556e94a20e10 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4359e640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0127e0 tx=0x7f3c3c01ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7cbc5b1c72b8c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:44.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c016040 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f3c3c01b460 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c3c01bb10 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c3c004e50 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 10 client.15564 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e9464e170 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c3c01bb10 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c3c023070 con 0x556e94a20e10 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 10 client.15564 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 20 client.15564 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a31280 0x556e94a33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 4 client.15564 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.021+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e94989920 con 0x556e94a31280 2024-04-22T06:37:44.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.022+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a31280 0x556e94a33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.022+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a31280 0x556e94a33670 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bfa420b06240cb6 server_cookie=108ea3b7c702cdd3 in_seq=0 out_seq=0 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.022+0000 7f3c2a7fc640 10 client.15564 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/680618530 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e94989920 con 0x556e94a31280 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c2a7fc640 10 client.15564 handle_command_reply: tid=0 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 client.15564 shutdown 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 msgr2=0x556e94a1e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0127e0 tx=0x7f3c3c01ccd0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 shutdown_connections 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a31280 0x556e94a33670 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a20e10 0x556e94a1e8c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a18290 0x556e94a1e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/680618530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a178e0 0x556e94a1de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 >> 172.21.15.105:0/680618530 conn(0x556e94a11fb0 msgr2=0x556e94a13d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c2a7fc640 0 client.15564 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 shutdown_connections 2024-04-22T06:37:44.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/680618530 wait complete. 2024-04-22T06:37:44.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.023+0000 7f3c4bdaf740 20 client.15564 trim_cache size 0 max 16384 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.040+0000 7f3c4bdaf740 1 Processor -- start 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.040+0000 7f3c4bdaf740 1 -- start start 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 0x556e94a2cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a20e10 0x556e94a10e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441afc90 con 0x556e94a18290 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441b17d0 con 0x556e94a178e0 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441b0360 con 0x556e94a20e10 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34450/0 (socket says 172.21.15.105:34450) 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3504605799 learned_addr learned my addr 172.21.15.105:0/3504605799 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 0x556e94a2cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3504605799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a20e10 msgr2=0x556e94a10e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a20e10 0x556e94a10e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3504605799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 msgr2=0x556e94a2cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 0x556e94a2cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 -- 172.21.15.105:0/3504605799 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e94a2d1b0 con 0x556e94a18290 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 0x556e94a2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0f0380 tx=0x7f3c2c0df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27999e348e5d4498 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.041+0000 7f3c38ff9640 1 -- 172.21.15.105:0/3504605799 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0bf0f0 con 0x556e94a18290 2024-04-22T06:37:44.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.042+0000 7f3c38ff9640 1 -- 172.21.15.105:0/3504605799 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c2c0f7040 con 0x556e94a18290 2024-04-22T06:37:44.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.042+0000 7f3c38ff9640 1 -- 172.21.15.105:0/3504605799 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0bb060 con 0x556e94a18290 2024-04-22T06:37:44.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.042+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/3504605799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 msgr2=0x556e94a1d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.042+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0f0380 tx=0x7f3c2c0df040 comp rx=0 tx=0).stop 2024-04-22T06:37:44.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/3504605799 shutdown_connections 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a20e10 0x556e94a10e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a18290 0x556e94a1d5f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/3504605799 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a178e0 0x556e94a2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/3504605799 >> 172.21.15.105:0/3504605799 conn(0x556e94a11fb0 msgr2=0x556e94a10780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/3504605799 shutdown_connections 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/3504605799 wait complete. 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- start start 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 Processor -- start 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a248d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.043+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441b1100 con 0x556e94a178e0 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441b0a30 con 0x556e94a20e10 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441adcb0 con 0x556e94a18290 2024-04-22T06:37:44.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 10 client.0 _ll_register_callbacks cb 0x7f3c3c009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34454/0 (socket says 172.21.15.105:34454) 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 -- 172.21.15.105:0/1879336568 learned_addr learned my addr 172.21.15.105:0/1879336568 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 -- 172.21.15.105:0/1879336568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 msgr2=0x556e94a296b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a296b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 -- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 msgr2=0x556e94a248d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 -- 172.21.15.105:0/1879336568 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e94a25100 con 0x556e94a178e0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c42d9d640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3c340058c0 tx=0x7f3c340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d515eed9d1fc22 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3402c070 con 0x556e94a178e0 2024-04-22T06:37:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f3c3401cd50 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c2c0f7040 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c34037050 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 10 client.15570 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556e94a97110 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.044+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c2c0f7040 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c34019040 con 0x556e94a178e0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 10 client.15570 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 10 client.15570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 20 client.15570 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a97810 0x556e94a99c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 4 client.15570 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556e94983790 con 0x556e94a97810 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a97810 0x556e94a99c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c3bfff640 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a97810 0x556e94a99c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=794795bc305282a3 server_cookie=644efa75e9003599 in_seq=0 out_seq=0 2024-04-22T06:37:44.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.045+0000 7f3c2a7fc640 10 client.15570 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.046+0000 7f3c2a7fc640 1 -- 172.21.15.105:0/1879336568 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556e94983790 con 0x556e94a97810 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.046+0000 7f3c2a7fc640 10 client.15570 handle_command_reply: tid=0 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.046+0000 7f3c4bdaf740 1 client.15570 shutdown 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 msgr2=0x556e94a29170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3c340058c0 tx=0x7f3c340046c0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 shutdown_connections 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556e94a97810 0x556e94a99c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c2a7fc640 0 client.15570 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556e94a20e10 0x556e94a248d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556e94a18290 0x556e94a296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 --2- 172.21.15.105:0/1879336568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556e94a178e0 0x556e94a29170 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 >> 172.21.15.105:0/1879336568 conn(0x556e94a11fb0 msgr2=0x556e94a13b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 shutdown_connections 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 1 -- 172.21.15.105:0/1879336568 wait complete. 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c4bdaf740 20 client.15570 trim_cache size 0 max 16384 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stdout: "return_code": 0, 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_tag": "01833dd3-3372-45e9-bd18-211e28cda9ac", 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": "asynchronous" 2024-04-22T06:37:44.051 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:44.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 msgr2=0x7f3c300a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 0x7f3c300a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x556e94a13020 tx=0x7f3c3400a2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 msgr2=0x7f3c441a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c036040 tx=0x7f3c2c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 shutdown_connections 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3c300a3ec0 0x7f3c300a6380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3c4419a8b0 0x7f3c4419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c441094d0 0x7f3c441a0bf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 --2- 172.21.15.105:0/3333433475 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c44104400 0x7f3c441a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 >> 172.21.15.105:0/3333433475 conn(0x7f3c440ffce0 msgr2=0x7f3c441010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.047+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 shutdown_connections 2024-04-22T06:37:44.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.048+0000 7f3c490e0640 1 -- 172.21.15.105:0/3333433475 wait complete. 2024-04-22T06:37:44.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:37:44.057 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:37:44.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.161+0000 7f5ccdf85640 1 -- 172.21.15.105:0/131572900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 msgr2=0x7f5cc8101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.161+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/131572900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8101460 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009900 tx=0x7f5cbc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.161+0000 7f5ccdf85640 1 -- 172.21.15.105:0/131572900 shutdown_connections 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.161+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/131572900 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.161+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/131572900 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc8101a30 0x7f5cc8105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/131572900 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8101460 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- 172.21.15.105:0/131572900 >> 172.21.15.105:0/131572900 conn(0x7f5cc8078000 msgr2=0x7f5cc8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- 172.21.15.105:0/131572900 shutdown_connections 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- 172.21.15.105:0/131572900 wait complete. 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 Processor -- start 2024-04-22T06:37:44.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- start start 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc819bec0 0x7f5cc819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81110e0 con 0x7f5cc8101080 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc8110f60 con 0x7f5cc8106150 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.162+0000 7f5ccdf85640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc8111260 con 0x7f5cc819bec0 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34478/0 (socket says 172.21.15.105:34478) 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3690683173 learned_addr learned my addr 172.21.15.105:0/3690683173 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc7fff640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc819bec0 0x7f5cc819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3690683173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc819bec0 msgr2=0x7f5cc819c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc819bec0 0x7f5cc819c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3690683173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 msgr2=0x7f5cc8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3690683173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc819c9b0 con 0x7f5cc8101080 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc000c00 tx=0x7f5cbc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc4ff9640 1 -- 172.21.15.105:0/3690683173 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc043020 con 0x7f5cc8101080 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc4ff9640 1 -- 172.21.15.105:0/3690683173 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cbc02cbb0 con 0x7f5cc8101080 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5cc4ff9640 1 -- 172.21.15.105:0/3690683173 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc003ac0 con 0x7f5cc8101080 2024-04-22T06:37:44.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc819cc40 con 0x7f5cc8101080 2024-04-22T06:37:44.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.163+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc81a4410 con 0x7f5cc8101080 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.165+0000 7f5cc4ff9640 1 -- 172.21.15.105:0/3690683173 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f5cbc046020 con 0x7f5cc8101080 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.166+0000 7f5cc4ff9640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 0x7f5cb00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.166+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 0x7f5cb00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.166+0000 7f5cc4ff9640 1 -- 172.21.15.105:0/3690683173 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5cbc0f2af0 con 0x7f5cc8101080 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.166+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 0x7f5cb00a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8005d00 tx=0x7f5cb8009550 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 Processor -- start 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 -- start start 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a57d0191e0 con 0x55a57d0b4ea0 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a57cfb6390 con 0x55a57d0bda80 2024-04-22T06:37:44.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a57d0151a0 con 0x55a57d0b4580 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34490/0 (socket says 172.21.15.105:34490) 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/1259024467 learned_addr learned my addr 172.21.15.105:0/1259024467 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc7fff640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/1259024467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 msgr2=0x55a57d0b4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0b4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/1259024467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 msgr2=0x55a57d0bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0bfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/1259024467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57cdf57d0 con 0x55a57d0b4ea0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.167+0000 7f5cc7fff640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.168+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800ee50 tx=0x7f5cb8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0285ee37479921 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.168+0000 7f5cadffb640 1 -- 172.21.15.105:0/1259024467 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb801e640 con 0x55a57d0b4ea0 2024-04-22T06:37:44.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.168+0000 7f5cadffb640 1 -- 172.21.15.105:0/1259024467 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb8023070 con 0x55a57d0b4ea0 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.168+0000 7f5cadffb640 1 -- 172.21.15.105:0/1259024467 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8010a00 con 0x55a57d0b4ea0 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- 172.21.15.105:0/1259024467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 msgr2=0x55a57d0bd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800ee50 tx=0x7f5cb8011000 comp rx=0 tx=0).stop 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- 172.21.15.105:0/1259024467 shutdown_connections 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0bd540 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/1259024467 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0b4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- 172.21.15.105:0/1259024467 >> 172.21.15.105:0/1259024467 conn(0x55a57d0aecb0 msgr2=0x55a57d0af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- 172.21.15.105:0/1259024467 shutdown_connections 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- 172.21.15.105:0/1259024467 wait complete. 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 -- start start 2024-04-22T06:37:44.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.169+0000 7f5cd0b71740 1 Processor -- start 2024-04-22T06:37:44.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:43 smithi105 ceph-mon[27916]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:44.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:43 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2172411492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:44.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:43 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2394520362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.171 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:43 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2146998714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a57d0c0810 con 0x55a57d0b4580 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a57cfd22f0 con 0x55a57d0bda80 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55a57d0c0990 con 0x55a57d0b4ea0 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34494/0 (socket says 172.21.15.105:34494) 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3404087088 learned_addr learned my addr 172.21.15.105:0/3404087088 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 10 client.0 _ll_register_callbacks cb 0x55a57cfdf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3404087088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 msgr2=0x55a57d0c1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc7fff640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 msgr2=0x55a57d0c16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 -- 172.21.15.105:0/3404087088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57cd2b3f0 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0f08b0 tx=0x7f5cbc0be6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32637611f62c0e30 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5ca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0d8020 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5cbc003c60 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.170+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cb8023070 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cbc003e60 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 10 client.15582 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57cceb0f0 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb8023070 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cbc0f0410 con 0x55a57d0b4580 2024-04-22T06:37:44.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 10 client.15582 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 20 client.15582 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d0cfaf0 0x55a57d0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 4 client.15582 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a57d026c00 con 0x55a57d0cfaf0 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.171+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d0cfaf0 0x55a57d0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.172+0000 7f5ca6ffd640 10 client.15582 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.172+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d0cfaf0 0x55a57d0d1f20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2360963426c8ae9e server_cookie=2ece1bef7fe0550a in_seq=0 out_seq=0 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.172+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/3404087088 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a57d026c00 con 0x55a57d0cfaf0 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.172+0000 7f5ca6ffd640 10 client.15582 handle_command_reply: tid=0 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.172+0000 7f5cd0b71740 1 client.15582 shutdown 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 msgr2=0x55a57d0c0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0f08b0 tx=0x7f5cbc0be6e0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 shutdown_connections 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d0cfaf0 0x55a57d0d1f20 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5ca6ffd640 0 client.15582 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/3404087088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c0c50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 >> 172.21.15.105:0/3404087088 conn(0x55a57d0aecb0 msgr2=0x55a57d0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 shutdown_connections 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 1 -- 172.21.15.105:0/3404087088 wait complete. 2024-04-22T06:37:44.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.173+0000 7f5cd0b71740 20 client.15582 trim_cache size 0 max 16384 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 Processor -- start 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 -- start start 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81ab6e0 con 0x55a57d0b4ea0 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81ad220 con 0x55a57d0bda80 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81abdb0 con 0x55a57d0b4580 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cc6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cc6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34510/0 (socket says 172.21.15.105:34510) 2024-04-22T06:37:44.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.205+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/2546688124 learned_addr learned my addr 172.21.15.105:0/2546688124 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/2546688124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 msgr2=0x55a57d0d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/2546688124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 msgr2=0x55a57d0b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 -- 172.21.15.105:0/2546688124 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a57d0b4cd0 con 0x55a57d0b4ea0 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8013040 tx=0x7f5cb80105e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5b813ac598610c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cad7fa640 1 -- 172.21.15.105:0/2546688124 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8036030 con 0x55a57d0b4ea0 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cad7fa640 1 -- 172.21.15.105:0/2546688124 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb8023030 con 0x55a57d0b4ea0 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.206+0000 7f5cad7fa640 1 -- 172.21.15.105:0/2546688124 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8032330 con 0x55a57d0b4ea0 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- 172.21.15.105:0/2546688124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 msgr2=0x55a57d0d7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8013040 tx=0x7f5cb80105e0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- 172.21.15.105:0/2546688124 shutdown_connections 2024-04-22T06:37:44.210 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4ea0 0x55a57d0d7d10 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/2546688124 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4580 0x55a57d0d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- 172.21.15.105:0/2546688124 >> 172.21.15.105:0/2546688124 conn(0x55a57d0aecb0 msgr2=0x55a57d0b9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- 172.21.15.105:0/2546688124 shutdown_connections 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- 172.21.15.105:0/2546688124 wait complete. 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- start start 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 Processor -- start 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81acb50 con 0x55a57d0b4580 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81ac480 con 0x55a57d0bda80 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.207+0000 7f5cd0b71740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81a96c0 con 0x55a57d0b4ea0 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cd0b71740 10 client.0 _ll_register_callbacks cb 0x7f5cbc0be980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:44.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34516/0 (socket says 172.21.15.105:34516) 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 -- 172.21.15.105:0/329909727 learned_addr learned my addr 172.21.15.105:0/329909727 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 -- 172.21.15.105:0/329909727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 msgr2=0x55a57d0c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 -- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 msgr2=0x55a57d0c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 -- 172.21.15.105:0/329909727 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a57d0c31e0 con 0x55a57d0b4580 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cc77fe640 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005ec0 tx=0x7f5cbc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bae16e95648195 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5ca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc0d8040 con 0x55a57d0b4580 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5cbc0ba970 con 0x55a57d0b4580 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cb8023030 con 0x55a57d0b4580 2024-04-22T06:37:44.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5cbc0f0180 con 0x55a57d0b4580 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cd0b71740 10 client.15588 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a57d138520 con 0x55a57d0b4580 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.208+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb8023030 con 0x55a57d0b4580 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5cbc083b80 con 0x55a57d0b4580 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 10 client.15588 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 20 client.15588 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d138ac0 0x55a57d13af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 4 client.15588 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a57d0ba7d0 con 0x55a57d138ac0 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d138ac0 0x55a57d13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5ca6ffd640 10 client.15588 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.209+0000 7f5cc6ffd640 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d138ac0 0x55a57d13af40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38322636680dac32 server_cookie=f3c483aa0631d952 in_seq=0 out_seq=0 2024-04-22T06:37:44.213 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ca6ffd640 1 -- 172.21.15.105:0/329909727 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a57d0ba7d0 con 0x55a57d138ac0 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ca6ffd640 10 client.15588 handle_command_reply: tid=0 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 client.15588 shutdown 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 msgr2=0x55a57d0c70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005ec0 tx=0x7f5cbc005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 shutdown_connections 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55a57d138ac0 0x55a57d13af40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a57d0bda80 0x55a57d0c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a57d0b4ea0 0x55a57d0c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ca6ffd640 0 client.15588 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 --2- 172.21.15.105:0/329909727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a57d0b4580 0x55a57d0c70c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 >> 172.21.15.105:0/329909727 conn(0x55a57d0aecb0 msgr2=0x55a57d0c0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 shutdown_connections 2024-04-22T06:37:44.214 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 1 -- 172.21.15.105:0/329909727 wait complete. 2024-04-22T06:37:44.215 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5cd0b71740 20 client.15588 trim_cache size 0 max 16384 2024-04-22T06:37:44.215 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:43 smithi112 ceph-mon[31236]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:44.215 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:43 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2172411492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:44.215 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:43 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2394520362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.215 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:43 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2146998714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.215 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:44.215 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:37:44.215 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": { 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: "01833dd3-3372-45e9-bd18-211e28cda9ac": { 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: "path": "/", 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: "tag": "01833dd3-3372-45e9-bd18-211e28cda9ac", 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: "options": "recursive,force" 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:37:44.216 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 msgr2=0x7f5cb00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 0x7f5cb00a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8005d00 tx=0x7f5cb8009550 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 msgr2=0x7f5cc8197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc000c00 tx=0x7f5cbc02c6c0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 shutdown_connections 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5cb00a3e40 0x7f5cb00a6300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5cc819bec0 0x7f5cc819c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5cc8106150 0x7f5cc8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 --2- 172.21.15.105:0/3690683173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cc8101080 0x7f5cc8197600 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.217 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.210+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 >> 172.21.15.105:0/3690683173 conn(0x7f5cc8078000 msgr2=0x7f5cc80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.211+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 shutdown_connections 2024-04-22T06:37:44.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.211+0000 7f5ccdf85640 1 -- 172.21.15.105:0/3690683173 wait complete. 2024-04-22T06:37:44.219 INFO:tasks.cephfs.filesystem:scrub status for tag:01833dd3-3372-45e9-bd18-211e28cda9ac - {'path': '/', 'tag': '01833dd3-3372-45e9-bd18-211e28cda9ac', 'options': 'recursive,force'} 2024-04-22T06:37:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:43 smithi177 ceph-mon[31346]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:37:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:43 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2172411492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:43 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2394520362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:43 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2146998714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:44.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.248+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd880062390 con 0x7fd86c0a3ee0 2024-04-22T06:37:44.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.321+0000 7fd87cff9640 1 -- 172.21.15.105:0/4191359411 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd880062390 con 0x7fd86c0a3ee0 2024-04-22T06:37:44.320 INFO:teuthology.orchestra.run.smithi105.stdout:Schedule set for path / 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 msgr2=0x7fd86c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 0x7fd86c0a63a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd8800fff10 tx=0x7fd86800a2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 msgr2=0x7fd880198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd870005e30 tx=0x7fd870003c20 comp rx=0 tx=0).stop 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 shutdown_connections 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd86c0a3ee0 0x7fd86c0a63a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd88010c520 0x7fd88019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd880103b00 0x7fd880198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 --2- 172.21.15.105:0/4191359411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd8801031e0 0x7fd880198710 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 >> 172.21.15.105:0/4191359411 conn(0x7fd880077fd0 msgr2=0x7fd8800ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 shutdown_connections 2024-04-22T06:37:44.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.323+0000 7fd885fa4640 1 -- 172.21.15.105:0/4191359411 wait complete. 2024-04-22T06:37:44.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 -- 172.21.15.105:0/2901641188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244106f70 msgr2=0x7f3244107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 --2- 172.21.15.105:0/2901641188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244106f70 0x7f3244107350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3240009930 tx=0x7f324002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 -- 172.21.15.105:0/2901641188 shutdown_connections 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 --2- 172.21.15.105:0/2901641188 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244101af0 0x7f324410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 --2- 172.21.15.105:0/2901641188 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f32441015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 --2- 172.21.15.105:0/2901641188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244106f70 0x7f3244107350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 -- 172.21.15.105:0/2901641188 >> 172.21.15.105:0/2901641188 conn(0x7f3244077ba0 msgr2=0x7f3244077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 -- 172.21.15.105:0/2901641188 shutdown_connections 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.475+0000 7f324c352640 1 -- 172.21.15.105:0/2901641188 wait complete. 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 Processor -- start 2024-04-22T06:37:44.476 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 -- start start 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f3244197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244106f70 0x7f324419be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3244111080 con 0x7f3244101af0 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3244110f00 con 0x7f3244106f70 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324c352640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3244111200 con 0x7f3244101130 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34536/0 (socket says 172.21.15.105:34536) 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 -- 172.21.15.105:0/899879719 learned_addr learned my addr 172.21.15.105:0/899879719 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324a8ef640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244106f70 0x7f324419be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324a0ee640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f3244197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 -- 172.21.15.105:0/899879719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 msgr2=0x7f3244197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f3244197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 -- 172.21.15.105:0/899879719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244106f70 msgr2=0x7f324419be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244106f70 0x7f324419be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 -- 172.21.15.105:0/899879719 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324419c570 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f324a0ee640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f3244197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.476+0000 7f32498ed640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f323400c920 tx=0x7f323400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.477+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234007c70 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.477+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3234007e10 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.477+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3234018820 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.477+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324419c860 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.477+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32441a40f0 con 0x7f3244101af0 2024-04-22T06:37:44.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.478+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32441025a0 con 0x7f3244101af0 2024-04-22T06:37:44.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.479+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f323400f450 con 0x7f3244101af0 2024-04-22T06:37:44.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.479+0000 7f323b7fe640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.479+0000 7f324a0ee640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.479+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f32340d0b70 con 0x7f3244101af0 2024-04-22T06:37:44.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.480+0000 7f324a0ee640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 0x7f32300a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3240006010 tx=0x7f324002c860 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.484+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3234014030 con 0x7f3244101af0 2024-04-22T06:37:44.680 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.682+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f3244062390 con 0x7f32300a3ee0 2024-04-22T06:37:44.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.742+0000 7f323b7fe640 1 -- 172.21.15.105:0/899879719 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f3244062390 con 0x7f32300a3ee0 2024-04-22T06:37:44.741 INFO:teuthology.orchestra.run.smithi105.stdout:Retention added to path / 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 msgr2=0x7f32300a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 0x7f32300a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3240006010 tx=0x7f324002c860 comp rx=0 tx=0).stop 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 msgr2=0x7f3244197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f323400c920 tx=0x7f323400cde0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 shutdown_connections 2024-04-22T06:37:44.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f32300a3ee0 0x7f32300a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3244106f70 0x7f324419be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3244101af0 0x7f3244197ac0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 --2- 172.21.15.105:0/899879719 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3244101130 0x7f3244197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.744+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 >> 172.21.15.105:0/899879719 conn(0x7f3244077ba0 msgr2=0x7f32440ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.745+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 shutdown_connections 2024-04-22T06:37:44.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.745+0000 7f324c352640 1 -- 172.21.15.105:0/899879719 wait complete. 2024-04-22T06:37:44.753 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:37:44.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.853+0000 7f5708c3e640 1 -- 172.21.15.105:0/2039330097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57041011a0 msgr2=0x7f5704101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.853+0000 7f5708c3e640 1 --2- 172.21.15.105:0/2039330097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57041011a0 0x7f5704101620 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009900 tx=0x7f56f802bc80 comp rx=0 tx=0).stop 2024-04-22T06:37:44.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 -- 172.21.15.105:0/2039330097 shutdown_connections 2024-04-22T06:37:44.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 --2- 172.21.15.105:0/2039330097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704101b60 0x7f570410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 --2- 172.21.15.105:0/2039330097 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57041011a0 0x7f5704101620 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 --2- 172.21.15.105:0/2039330097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5704106fb0 0x7f5704107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 -- 172.21.15.105:0/2039330097 >> 172.21.15.105:0/2039330097 conn(0x7f5704076c50 msgr2=0x7f5704077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:44.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 -- 172.21.15.105:0/2039330097 shutdown_connections 2024-04-22T06:37:44.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 -- 172.21.15.105:0/2039330097 wait complete. 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 Processor -- start 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.854+0000 7f5708c3e640 1 -- start start 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57041011a0 0x7f57041994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704106fb0 0x7f57041a4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5704111060 con 0x7f5704101b60 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5704110ee0 con 0x7f5704106fb0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f57041111e0 con 0x7f57041011a0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40656/0 (socket says 172.21.15.105:40656) 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 -- 172.21.15.105:0/3589805680 learned_addr learned my addr 172.21.15.105:0/3589805680 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f570259c640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57041011a0 0x7f57041994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 -- 172.21.15.105:0/3589805680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57041011a0 msgr2=0x7f57041994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5702d9d640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704106fb0 0x7f57041a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57041011a0 0x7f57041994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 -- 172.21.15.105:0/3589805680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704106fb0 msgr2=0x7f57041a4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704106fb0 0x7f57041a4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 -- 172.21.15.105:0/3589805680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57041a4a60 con 0x7f5704101b60 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5701d9b640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005940 tx=0x7f56f802c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8043020 con 0x7f5704101b60 2024-04-22T06:37:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56f802cbd0 con 0x7f5704101b60 2024-04-22T06:37:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57041a4c30 con 0x7f5704101b60 2024-04-22T06:37:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f8003bf0 con 0x7f5704101b60 2024-04-22T06:37:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.855+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57041a50f0 con 0x7f5704101b60 2024-04-22T06:37:44.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.856+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5704102550 con 0x7f5704101b60 2024-04-22T06:37:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.857+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f56f8046020 con 0x7f5704101b60 2024-04-22T06:37:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.858+0000 7f56f37fe640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 0x7f56ec0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.858+0000 7f570259c640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 0x7f56ec0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.858+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f56f80f2a90 con 0x7f5704101b60 2024-04-22T06:37:44.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.858+0000 7f570259c640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 0x7f56ec0a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56f4005d00 tx=0x7f56f400b410 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:44.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:44.862+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56f804d020 con 0x7f5704101b60 2024-04-22T06:37:45.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.067+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f5704062390 con 0x7f56ec0a3ee0 2024-04-22T06:37:45.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.075+0000 7f56f37fe640 1 -- 172.21.15.105:0/3589805680 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f5704062390 con 0x7f56ec0a3ee0 2024-04-22T06:37:45.073 INFO:teuthology.orchestra.run.smithi105.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:37:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-22T06:37:45.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 msgr2=0x7f56ec0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.075 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 0x7f56ec0a63a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56f4005d00 tx=0x7f56f400b410 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 msgr2=0x7f57041999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f56f8005940 tx=0x7f56f802c7d0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 shutdown_connections 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f56ec0a3ee0 0x7f56ec0a63a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5704106fb0 0x7f57041a4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5704101b60 0x7f57041999e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 --2- 172.21.15.105:0/3589805680 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f57041011a0 0x7f57041994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 >> 172.21.15.105:0/3589805680 conn(0x7f5704076c50 msgr2=0x7f57040fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 shutdown_connections 2024-04-22T06:37:45.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.077+0000 7f5708c3e640 1 -- 172.21.15.105:0/3589805680 wait complete. 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: scrub queued for path: / 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: scrub summary: active paths [/] 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.085 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:44 smithi105 ceph-mon[27916]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/4111048260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc06b3a0 msgr2=0x7f1acc106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/4111048260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc06b3a0 0x7f1acc106b80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009900 tx=0x7f1ac002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/4111048260 shutdown_connections 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/4111048260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc107260 0x7f1acc10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/4111048260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc06b3a0 0x7f1acc106b80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/4111048260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc06add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/4111048260 >> 172.21.15.105:0/4111048260 conn(0x7f1acc076b80 msgr2=0x7f1acc076f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/4111048260 shutdown_connections 2024-04-22T06:37:45.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/4111048260 wait complete. 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.189+0000 7f1ad1a4e640 1 Processor -- start 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 -- start start 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 0x7f1acc19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc1121a0 con 0x7f1acc107260 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc112020 con 0x7f1acc06a9f0 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc112320 con 0x7f1acc19cf20 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40674/0 (socket says 172.21.15.105:40674) 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 -- 172.21.15.105:0/1647964950 learned_addr learned my addr 172.21.15.105:0/1647964950 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1acb7fe640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 0x7f1acc19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1acaffd640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 -- 172.21.15.105:0/1647964950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 msgr2=0x7f1acc19d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 0x7f1acc19d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 -- 172.21.15.105:0/1647964950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 msgr2=0x7f1acc198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 -- 172.21.15.105:0/1647964950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc19da10 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1acb7fe640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 0x7f1acc19d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1acaffd640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1aca7fc640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0036040 tx=0x7f1ac002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0043020 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac002cbe0 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc19dca0 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.190+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0003bf0 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.191+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc1a54c0 con 0x7f1acc107260 2024-04-22T06:37:45.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.192+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1acc103660 con 0x7f1acc107260 2024-04-22T06:37:45.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.193+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f1ac0046020 con 0x7f1acc107260 2024-04-22T06:37:45.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.193+0000 7f1ad0a4c640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 0x7f1ab40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.194+0000 7f1acaffd640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 0x7f1ab40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.194+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f1ac00f2c30 con 0x7f1acc107260 2024-04-22T06:37:45.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.194+0000 7f1acaffd640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 0x7f1ab40a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0058c0 tx=0x7f1abc005d30 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.198+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ac004d020 con 0x7f1acc107260 2024-04-22T06:37:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: scrub queued for path: / 2024-04-22T06:37:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: scrub summary: active paths [/] 2024-04-22T06:37:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:44 smithi112 ceph-mon[31236]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: scrub queued for path: / 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: scrub summary: active paths [/] 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:44 smithi177 ceph-mon[31346]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:45.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.395+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1acc06b3a0 con 0x7f1ab40a3ee0 2024-04-22T06:37:45.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.402+0000 7f1ad0a4c640 1 -- 172.21.15.105:0/1647964950 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1acc06b3a0 con 0x7f1ab40a3ee0 2024-04-22T06:37:45.400 INFO:teuthology.orchestra.run.smithi105.stdout:/ 1m 6m3h 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 msgr2=0x7f1ab40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 0x7f1ab40a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0058c0 tx=0x7f1abc005d30 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 msgr2=0x7f1acc198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0036040 tx=0x7f1ac002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 shutdown_connections 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f1ab40a3ee0 0x7f1ab40a63a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1acc19cf20 0x7f1acc19d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1acc107260 0x7f1acc198ba0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 --2- 172.21.15.105:0/1647964950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1acc06a9f0 0x7f1acc198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 >> 172.21.15.105:0/1647964950 conn(0x7f1acc076b80 msgr2=0x7f1acc0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 shutdown_connections 2024-04-22T06:37:45.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.404+0000 7f1ad1a4e640 1 -- 172.21.15.105:0/1647964950 wait complete. 2024-04-22T06:37:45.413 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-22T06:37:45.438 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-22T06:37:45.449 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-22T06:37:45.515 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:37:45.515 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-22T06:37:45.518 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-22T06:37:45.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:45.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.631+0000 7f5dd041f640 1 -- 172.21.15.105:0/484919223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 msgr2=0x7f5dc8101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.631+0000 7f5dd041f640 1 --2- 172.21.15.105:0/484919223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8101620 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5db8009930 tx=0x7f5db802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- 172.21.15.105:0/484919223 shutdown_connections 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- 172.21.15.105:0/484919223 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 0x7f5dc810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- 172.21.15.105:0/484919223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8101620 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- 172.21.15.105:0/484919223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 0x7f5dc8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- 172.21.15.105:0/484919223 >> 172.21.15.105:0/484919223 conn(0x7f5dc8077e30 msgr2=0x7f5dc8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- 172.21.15.105:0/484919223 shutdown_connections 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- 172.21.15.105:0/484919223 wait complete. 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 Processor -- start 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- start start 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 0x7f5dc8198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 0x7f5dc819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc8112240 con 0x7f5dc81011a0 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81120c0 con 0x7f5dc8105e30 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dd041f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc81123c0 con 0x7f5dc8101b60 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dce1bb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dce1bb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40698/0 (socket says 172.21.15.105:40698) 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.632+0000 7f5dce1bb640 1 -- 172.21.15.105:0/4151155186 learned_addr learned my addr 172.21.15.105:0/4151155186 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dcd9ba640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 0x7f5dc8198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce9bc640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 0x7f5dc819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 -- 172.21.15.105:0/4151155186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 msgr2=0x7f5dc8198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 0x7f5dc8198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 -- 172.21.15.105:0/4151155186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 msgr2=0x7f5dc819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 0x7f5dc819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 -- 172.21.15.105:0/4151155186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc819d700 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dce1bb640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c890 tx=0x7f5dc400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4004220 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dc40043c0 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc819d9f0 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4004a00 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.633+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc81a5280 con 0x7f5dc81011a0 2024-04-22T06:37:45.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.634+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc8068de0 con 0x7f5dc81011a0 2024-04-22T06:37:45.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.635+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f5dc40079b0 con 0x7f5dc81011a0 2024-04-22T06:37:45.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.636+0000 7f5dbf7fe640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 0x7f5db40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.636+0000 7f5dcd9ba640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 0x7f5db40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.636+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5dc40d0b90 con 0x7f5dc81011a0 2024-04-22T06:37:45.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.636+0000 7f5dcd9ba640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 0x7f5db40a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5db8005ab0 tx=0x7f5db8005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.639+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dc4014030 con 0x7f5dc81011a0 2024-04-22T06:37:45.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.817+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dc81093b0 con 0x7f5dc81011a0 2024-04-22T06:37:45.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.818+0000 7f5dbf7fe640 1 -- 172.21.15.105:0/4151155186 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7f5dc409d050 con 0x7f5dc81011a0 2024-04-22T06:37:45.817 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:45.817 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 msgr2=0x7f5db40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 0x7f5db40a6300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5db8005ab0 tx=0x7f5db8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 msgr2=0x7f5dc8198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c890 tx=0x7f5dc400cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 shutdown_connections 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5db40a3e40 0x7f5db40a6300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5dc8105e30 0x7f5dc819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5dc8101b60 0x7f5dc8198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 --2- 172.21.15.105:0/4151155186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5dc81011a0 0x7f5dc8198760 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 >> 172.21.15.105:0/4151155186 conn(0x7f5dc8077e30 msgr2=0x7f5dc80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 shutdown_connections 2024-04-22T06:37:45.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.820+0000 7f5dd041f640 1 -- 172.21.15.105:0/4151155186 wait complete. 2024-04-22T06:37:45.830 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:45.830 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:45.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 -- 172.21.15.105:0/2227491084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01021e0 msgr2=0x7ff5b01025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/2227491084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01021e0 0x7ff5b01025c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009900 tx=0x7ff5a002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 -- 172.21.15.105:0/2227491084 shutdown_connections 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/2227491084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b01072b0 0x7ff5b010ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/2227491084 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b0102b90 0x7ff5b0106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/2227491084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01021e0 0x7ff5b01025c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 -- 172.21.15.105:0/2227491084 >> 172.21.15.105:0/2227491084 conn(0x7ff5b00fda60 msgr2=0x7ff5b00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 -- 172.21.15.105:0/2227491084 shutdown_connections 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.933+0000 7ff5b72d2640 1 -- 172.21.15.105:0/2227491084 wait complete. 2024-04-22T06:37:45.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 Processor -- start 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 -- start start 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 0x7ff5b019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 0x7ff5b0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b0112210 con 0x7ff5b01072b0 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b0112090 con 0x7ff5b0198630 2024-04-22T06:37:45.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b72d2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b0112390 con 0x7ff5b01021e0 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40712/0 (socket says 172.21.15.105:40712) 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 -- 172.21.15.105:0/3984110022 learned_addr learned my addr 172.21.15.105:0/3984110022 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b586f640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 0x7ff5b0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b506e640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 0x7ff5b019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 -- 172.21.15.105:0/3984110022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 msgr2=0x7ff5b019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 0x7ff5b019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 -- 172.21.15.105:0/3984110022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 msgr2=0x7ff5b0198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 0x7ff5b0198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 -- 172.21.15.105:0/3984110022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b0199150 con 0x7ff5b01072b0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b506e640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 0x7ff5b019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b586f640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 0x7ff5b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.934+0000 7ff5b486d640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00e9f0 tx=0x7ff5ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.935+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00ccc0 con 0x7ff5b01072b0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.935+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5ac0054d0 con 0x7ff5b01072b0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.935+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac010430 con 0x7ff5b01072b0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.935+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b0199440 con 0x7ff5b01072b0 2024-04-22T06:37:45.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.935+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b01a55f0 con 0x7ff5b01072b0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.936+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b00691c0 con 0x7ff5b01072b0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.937+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7ff5ac0046c0 con 0x7ff5b01072b0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.937+0000 7ff5a67fc640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 0x7ff59c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.937+0000 7ff5b506e640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 0x7ff59c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.937+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7ff5ac0d1050 con 0x7ff5b01072b0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.938+0000 7ff5b506e640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 0x7ff59c0a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005ab0 tx=0x7ff5a0005a20 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:45.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:45.941+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5ac053ff0 con 0x7ff5b01072b0 2024-04-22T06:37:46.123 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.125+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff5b010a4e0 con 0x7ff5b01072b0 2024-04-22T06:37:46.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.125+0000 7ff5a67fc640 1 -- 172.21.15.105:0/3984110022 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7ff5ac09d5b0 con 0x7ff5b01072b0 2024-04-22T06:37:46.124 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 msgr2=0x7ff59c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 0x7ff59c0a6300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0005ab0 tx=0x7ff5a0005a20 comp rx=0 tx=0).stop 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 msgr2=0x7ff5b019e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00e9f0 tx=0x7ff5ac00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 shutdown_connections 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff59c0a3e40 0x7ff59c0a6300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5b0198630 0x7ff5b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5b01072b0 0x7ff5b019e970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 --2- 172.21.15.105:0/3984110022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5b01021e0 0x7ff5b019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 >> 172.21.15.105:0/3984110022 conn(0x7ff5b00fda60 msgr2=0x7ff5b00fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 shutdown_connections 2024-04-22T06:37:46.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.127+0000 7ff5b72d2640 1 -- 172.21.15.105:0/3984110022 wait complete. 2024-04-22T06:37:46.135 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:45 smithi105 ceph-mon[27916]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-22T06:37:46.136 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:45 smithi105 ceph-mon[27916]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.136 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:45 smithi105 ceph-mon[27916]: scrub summary: idle 2024-04-22T06:37:46.136 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:45 smithi105 ceph-mon[27916]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.136 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:45 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4151155186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:46.137 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:46.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 -- 172.21.15.105:0/676112893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d4102bc0 msgr2=0x7f52d4106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 --2- 172.21.15.105:0/676112893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d4102bc0 0x7f52d4106c00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009900 tx=0x7f52c402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 -- 172.21.15.105:0/676112893 shutdown_connections 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 --2- 172.21.15.105:0/676112893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d41072e0 0x7f52d410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 --2- 172.21.15.105:0/676112893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d4102bc0 0x7f52d4106c00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 --2- 172.21.15.105:0/676112893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d4102210 0x7f52d41025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.239+0000 7f52da785640 1 -- 172.21.15.105:0/676112893 >> 172.21.15.105:0/676112893 conn(0x7f52d40fdab0 msgr2=0x7f52d40ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- 172.21.15.105:0/676112893 shutdown_connections 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- 172.21.15.105:0/676112893 wait complete. 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 Processor -- start 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- start start 2024-04-22T06:37:46.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 0x7f52d419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d41986a0 0x7f52d4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4112280 con 0x7f52d41072e0 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4112100 con 0x7f52d41986a0 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.240+0000 7f52da785640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f52d4112400 con 0x7f52d4102210 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40736/0 (socket says 172.21.15.105:40736) 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 -- 172.21.15.105:0/2838801498 learned_addr learned my addr 172.21.15.105:0/2838801498 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d8d22640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d41986a0 0x7f52d4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d3fff640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 0x7f52d419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 -- 172.21.15.105:0/2838801498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 msgr2=0x7f52d419e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 0x7f52d419e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 -- 172.21.15.105:0/2838801498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d41986a0 msgr2=0x7f52d4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d41986a0 0x7f52d4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 -- 172.21.15.105:0/2838801498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d41992a0 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d3fff640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 0x7f52d419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d37fe640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f52c4005ec0 tx=0x7f52c402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4043020 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d4199530 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52c40052b0 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4003d40 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.241+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d41a56d0 con 0x7f52d41072e0 2024-04-22T06:37:46.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.242+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d41036e0 con 0x7f52d41072e0 2024-04-22T06:37:46.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:45 smithi177 ceph-mon[31346]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-22T06:37:46.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:45 smithi177 ceph-mon[31346]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:45 smithi177 ceph-mon[31346]: scrub summary: idle 2024-04-22T06:37:46.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:45 smithi177 ceph-mon[31346]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.244 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:45 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4151155186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:46.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.243+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f52c4046020 con 0x7f52d41072e0 2024-04-22T06:37:46.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.244+0000 7f52d17fa640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 0x7f52bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.244+0000 7f52d3fff640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 0x7f52bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.244+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f52c40f2ca0 con 0x7f52d41072e0 2024-04-22T06:37:46.245 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.244+0000 7f52d3fff640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 0x7f52bc0a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52c800baa0 tx=0x7f52c8005d90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.246 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.248+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f52c404d020 con 0x7f52d41072e0 2024-04-22T06:37:46.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.422+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f52d410a510 con 0x7f52d41072e0 2024-04-22T06:37:46.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.423+0000 7f52d17fa640 1 -- 172.21.15.105:0/2838801498 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f52c40bf270 con 0x7f52d41072e0 2024-04-22T06:37:46.421 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 msgr2=0x7f52bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 0x7f52bc0a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52c800baa0 tx=0x7f52c8005d90 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 msgr2=0x7f52d419eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f52c4005ec0 tx=0x7f52c402ca10 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 shutdown_connections 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f52bc0a3e40 0x7f52bc0a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f52d41986a0 0x7f52d4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52d41072e0 0x7f52d419eb20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 --2- 172.21.15.105:0/2838801498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f52d4102210 0x7f52d419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 >> 172.21.15.105:0/2838801498 conn(0x7f52d40fdab0 msgr2=0x7f52d40ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 shutdown_connections 2024-04-22T06:37:46.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.425+0000 7f52da785640 1 -- 172.21.15.105:0/2838801498 wait complete. 2024-04-22T06:37:46.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:45 smithi112 ceph-mon[31236]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.4 KiB/s wr, 26 op/s 2024-04-22T06:37:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:45 smithi112 ceph-mon[31236]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:45 smithi112 ceph-mon[31236]: scrub summary: idle 2024-04-22T06:37:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:45 smithi112 ceph-mon[31236]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:37:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:45 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4151155186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.535+0000 7f6c1147a640 1 -- 172.21.15.105:0/3593613237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c103af0 msgr2=0x7f6c0c10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.535+0000 7f6c1147a640 1 --2- 172.21.15.105:0/3593613237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c103af0 0x7f6c0c10bfd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009900 tx=0x7f6c0002bd60 comp rx=0 tx=0).stop 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- 172.21.15.105:0/3593613237 shutdown_connections 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- 172.21.15.105:0/3593613237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c10c510 0x7f6c0c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- 172.21.15.105:0/3593613237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c103af0 0x7f6c0c10bfd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- 172.21.15.105:0/3593613237 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- 172.21.15.105:0/3593613237 >> 172.21.15.105:0/3593613237 conn(0x7f6c0c077f50 msgr2=0x7f6c0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- 172.21.15.105:0/3593613237 shutdown_connections 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- 172.21.15.105:0/3593613237 wait complete. 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 Processor -- start 2024-04-22T06:37:46.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- start start 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 0x7f6c0c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 0x7f6c0c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.536+0000 7f6c1147a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c1121a0 con 0x7f6c0c1031d0 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c1147a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c112020 con 0x7f6c0c103af0 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c1147a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c112320 con 0x7f6c0c10c510 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40760/0 (socket says 172.21.15.105:40760) 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 -- 172.21.15.105:0/974187636 learned_addr learned my addr 172.21.15.105:0/974187636 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0a7fc640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 0x7f6c0c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0b7fe640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 0x7f6c0c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 -- 172.21.15.105:0/974187636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 msgr2=0x7f6c0c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 0x7f6c0c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 -- 172.21.15.105:0/974187636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 msgr2=0x7f6c0c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 0x7f6c0c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 -- 172.21.15.105:0/974187636 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c198d30 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0a7fc640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 0x7f6c0c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0b7fe640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 0x7f6c0c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c0affd640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00d770 tx=0x7f6bfc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc017070 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bfc0054d0 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c199020 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc004c30 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.537+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1a52c0 con 0x7f6c0c1031d0 2024-04-22T06:37:46.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.538+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c068d70 con 0x7f6c0c1031d0 2024-04-22T06:37:46.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.539+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f6bfc004e10 con 0x7f6c0c1031d0 2024-04-22T06:37:46.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.540+0000 7f6bf3fff640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 0x7f6bf40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.540+0000 7f6c0a7fc640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 0x7f6bf40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.541 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.540+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f6bfc0d05c0 con 0x7f6c0c1031d0 2024-04-22T06:37:46.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.540+0000 7f6c0a7fc640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 0x7f6bf40a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c1996f0 tx=0x7f6c00005a20 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.544 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.545+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bfc09ca80 con 0x7f6c0c1031d0 2024-04-22T06:37:46.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.721+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c0c02d5c0 con 0x7f6c0c1031d0 2024-04-22T06:37:46.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.722+0000 7f6bf3fff640 1 -- 172.21.15.105:0/974187636 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f6bfc09c380 con 0x7f6c0c1031d0 2024-04-22T06:37:46.722 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:46.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 msgr2=0x7f6bf40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.723 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 0x7f6bf40a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c1996f0 tx=0x7f6c00005a20 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 msgr2=0x7f6c0c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00d770 tx=0x7f6bfc00dc30 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 shutdown_connections 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6bf40a3e40 0x7f6bf40a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6c0c10c510 0x7f6c0c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c0c103af0 0x7f6c0c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 --2- 172.21.15.105:0/974187636 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c0c1031d0 0x7f6c0c19e400 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 >> 172.21.15.105:0/974187636 conn(0x7f6c0c077f50 msgr2=0x7f6c0c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 shutdown_connections 2024-04-22T06:37:46.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.724+0000 7f6c1147a640 1 -- 172.21.15.105:0/974187636 wait complete. 2024-04-22T06:37:46.734 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 -- 172.21.15.105:0/3196032047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 msgr2=0x7fb1fc1083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 --2- 172.21.15.105:0/3196032047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc1083b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009930 tx=0x7fb1f802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 -- 172.21.15.105:0/3196032047 shutdown_connections 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 --2- 172.21.15.105:0/3196032047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1fc069aa0 0x7fb1fc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 --2- 172.21.15.105:0/3196032047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.832+0000 7fb2047ef640 1 --2- 172.21.15.105:0/3196032047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc1083b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- 172.21.15.105:0/3196032047 >> 172.21.15.105:0/3196032047 conn(0x7fb1fc077e30 msgr2=0x7fb1fc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- 172.21.15.105:0/3196032047 shutdown_connections 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- 172.21.15.105:0/3196032047 wait complete. 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 Processor -- start 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- start start 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1fc069aa0 0x7fb1fc1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1121f0 con 0x7fb1fc107fd0 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc112070 con 0x7fb1fc0690e0 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb2047ef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc112370 con 0x7fb1fc069aa0 2024-04-22T06:37:46.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb202d8c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb202d8c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40778/0 (socket says 172.21.15.105:40778) 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.833+0000 7fb202d8c640 1 -- 172.21.15.105:0/2359681399 learned_addr learned my addr 172.21.15.105:0/2359681399 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb20258b640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 -- 172.21.15.105:0/2359681399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1fc069aa0 msgr2=0x7fb1fc1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1fc069aa0 0x7fb1fc1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 -- 172.21.15.105:0/2359681399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 msgr2=0x7fb1fc101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 -- 172.21.15.105:0/2359681399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc1a6b70 con 0x7fb1fc107fd0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb20258b640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb202d8c640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400b970 tx=0x7fb1f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4019070 con 0x7fb1fc107fd0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc1a6e60 con 0x7fb1fc107fd0 2024-04-22T06:37:46.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1f4014440 con 0x7fb1fc107fd0 2024-04-22T06:37:46.837 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40075d0 con 0x7fb1fc107fd0 2024-04-22T06:37:46.837 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.834+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc1a73d0 con 0x7fb1fc107fd0 2024-04-22T06:37:46.837 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.835+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc068de0 con 0x7fb1fc107fd0 2024-04-22T06:37:46.837 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.836+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fb1f401c070 con 0x7fb1fc107fd0 2024-04-22T06:37:46.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.837+0000 7fb1f37fe640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 0x7fb1e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:46.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.837+0000 7fb20258b640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 0x7fb1e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:46.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.837+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fb1f40d0430 con 0x7fb1fc107fd0 2024-04-22T06:37:46.838 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.837+0000 7fb20258b640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 0x7fb1e80a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8036040 tx=0x7fb1f8005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:46.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:46.841+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1f409ca00 con 0x7fb1fc107fd0 2024-04-22T06:37:47.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.020+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1fc1027a0 con 0x7fb1fc107fd0 2024-04-22T06:37:47.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.022+0000 7fb1f37fe640 1 -- 172.21.15.105:0/2359681399 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7fb1f409c300 con 0x7fb1fc107fd0 2024-04-22T06:37:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:47.022 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:47.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 msgr2=0x7fb1e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.026 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 0x7fb1e80a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8036040 tx=0x7fb1f8005fd0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 msgr2=0x7fb1fc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400b970 tx=0x7fb1f400be30 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 shutdown_connections 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb1e80a3e40 0x7fb1e80a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1fc107fd0 0x7fb1fc101cf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb1fc069aa0 0x7fb1fc1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 --2- 172.21.15.105:0/2359681399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc0690e0 0x7fb1fc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 >> 172.21.15.105:0/2359681399 conn(0x7fb1fc077e30 msgr2=0x7fb1fc0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 shutdown_connections 2024-04-22T06:37:47.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.025+0000 7fb2047ef640 1 -- 172.21.15.105:0/2359681399 wait complete. 2024-04-22T06:37:47.036 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:47.037 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:47.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 -- 172.21.15.105:0/3839595079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101a40 msgr2=0x7f46c8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3839595079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101a40 0x7f46c8105a80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009900 tx=0x7f46bc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:47.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 -- 172.21.15.105:0/3839595079 shutdown_connections 2024-04-22T06:37:47.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3839595079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46c8106160 0x7f46c810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3839595079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101a40 0x7f46c8105a80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3839595079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c8101090 0x7f46c8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 -- 172.21.15.105:0/3839595079 >> 172.21.15.105:0/3839595079 conn(0x7f46c8077f50 msgr2=0x7f46c8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.138+0000 7f46ce82e640 1 -- 172.21.15.105:0/3839595079 shutdown_connections 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 -- 172.21.15.105:0/3839595079 wait complete. 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 Processor -- start 2024-04-22T06:37:47.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 -- start start 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46c8106160 0x7f46c8198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c819d010 0x7f46c819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46c81110a0 con 0x7f46c8101090 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46c8110f20 con 0x7f46c819d010 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46ce82e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46c8111220 con 0x7f46c8106160 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.139+0000 7f46c7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40800/0 (socket says 172.21.15.105:40800) 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 -- 172.21.15.105:0/3171880173 learned_addr learned my addr 172.21.15.105:0/3171880173 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46ccdcb640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c819d010 0x7f46c819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 -- 172.21.15.105:0/3171880173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46c8106160 msgr2=0x7f46c8198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46c8106160 0x7f46c8198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 -- 172.21.15.105:0/3171880173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c819d010 msgr2=0x7f46c819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c819d010 0x7f46c819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 -- 172.21.15.105:0/3171880173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c819db00 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c7fff640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f46b800d950 tx=0x7f46b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8017070 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c819ddf0 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b8009c30 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b800f620 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.140+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c81a55e0 con 0x7f46c8101090 2024-04-22T06:37:47.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.141+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c8068d70 con 0x7f46c8101090 2024-04-22T06:37:47.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.142+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f46b8010460 con 0x7f46c8101090 2024-04-22T06:37:47.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.143+0000 7f46c57fa640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 0x7f46b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.143+0000 7f46c77fe640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 0x7f46b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.143+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f46b80d0730 con 0x7f46c8101090 2024-04-22T06:37:47.143 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.143+0000 7f46c77fe640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 0x7f46b00a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0096c0 tx=0x7f46bc000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.147 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.148+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46b809cbf0 con 0x7f46c8101090 2024-04-22T06:37:47.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.324+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46c8101a40 con 0x7f46c8101090 2024-04-22T06:37:47.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.324+0000 7f46c57fa640 1 -- 172.21.15.105:0/3171880173 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f46b8009de0 con 0x7f46c8101090 2024-04-22T06:37:47.324 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 msgr2=0x7f46b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 0x7f46b00a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0096c0 tx=0x7f46bc000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 msgr2=0x7f46c8198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f46b800d950 tx=0x7f46b800de10 comp rx=0 tx=0).stop 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 shutdown_connections 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f46b00a3e40 0x7f46b00a6300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46c819d010 0x7f46c819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46c8106160 0x7f46c8198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 --2- 172.21.15.105:0/3171880173 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46c8101090 0x7f46c8198750 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 >> 172.21.15.105:0/3171880173 conn(0x7f46c8077f50 msgr2=0x7f46c80694f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 shutdown_connections 2024-04-22T06:37:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.326+0000 7f46ce82e640 1 -- 172.21.15.105:0/3171880173 wait complete. 2024-04-22T06:37:47.336 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3984110022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.336 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2838801498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.336 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/974187636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 -- 172.21.15.105:0/2418094171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 msgr2=0x7f664810eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 --2- 172.21.15.105:0/2418094171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664810eec0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6640009ca0 tx=0x7f664002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 -- 172.21.15.105:0/2418094171 shutdown_connections 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 --2- 172.21.15.105:0/2418094171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664810eec0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 --2- 172.21.15.105:0/2418094171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66481014a0 0x7f6648106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 --2- 172.21.15.105:0/2418094171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6648100af0 0x7f6648100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 -- 172.21.15.105:0/2418094171 >> 172.21.15.105:0/2418094171 conn(0x7f66480760e0 msgr2=0x7f66480764f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 -- 172.21.15.105:0/2418094171 shutdown_connections 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 -- 172.21.15.105:0/2418094171 wait complete. 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.440+0000 7f664e0ce640 1 Processor -- start 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 -- start start 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6648100af0 0x7f664819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66481985d0 0x7f6648198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6648112250 con 0x7f66481072b0 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f66481120d0 con 0x7f66481985d0 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f664e0ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f66481123d0 con 0x7f6648100af0 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40820/0 (socket says 172.21.15.105:40820) 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 -- 172.21.15.105:0/1856989731 learned_addr learned my addr 172.21.15.105:0/1856989731 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f66477fe640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6648100af0 0x7f664819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6647fff640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66481985d0 0x7f6648198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 -- 172.21.15.105:0/1856989731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6648100af0 msgr2=0x7f664819e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6648100af0 0x7f664819e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 -- 172.21.15.105:0/1856989731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66481985d0 msgr2=0x7f6648198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66481985d0 0x7f6648198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 -- 172.21.15.105:0/1856989731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66481990f0 con 0x7f66481072b0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.441+0000 7f6646ffd640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f663c00d740 tx=0x7f663c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.442+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c017070 con 0x7f66481072b0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.442+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66481993e0 con 0x7f66481072b0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.442+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f663c00de50 con 0x7f66481072b0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.442+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66481a5630 con 0x7f66481072b0 2024-04-22T06:37:47.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.442+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c004ce0 con 0x7f66481072b0 2024-04-22T06:37:47.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.443+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f664806a030 con 0x7f66481072b0 2024-04-22T06:37:47.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.444+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f663c004e80 con 0x7f66481072b0 2024-04-22T06:37:47.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.444+0000 7f6644ff9640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 0x7f66300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.444+0000 7f66477fe640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 0x7f66300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.446 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.444+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f663c0d0370 con 0x7f66481072b0 2024-04-22T06:37:47.446 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.445+0000 7f66477fe640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 0x7f66300a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f663800bbe0 tx=0x7f663800b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.449+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f663c09c830 con 0x7f66481072b0 2024-04-22T06:37:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3984110022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2838801498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/974187636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3984110022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2838801498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/974187636' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:47.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.628+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f664810a580 con 0x7f66481072b0 2024-04-22T06:37:47.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.628+0000 7f6644ff9640 1 -- 172.21.15.105:0/1856989731 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f663c09c130 con 0x7f66481072b0 2024-04-22T06:37:47.628 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:47.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 msgr2=0x7f66300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 0x7f66300a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f663800bbe0 tx=0x7f663800b390 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 msgr2=0x7f664819e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f663c00d740 tx=0x7f663c00dc00 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 shutdown_connections 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f66300a3e40 0x7f66300a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66481985d0 0x7f6648198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66481072b0 0x7f664819e960 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 --2- 172.21.15.105:0/1856989731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6648100af0 0x7f664819e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 >> 172.21.15.105:0/1856989731 conn(0x7f66480760e0 msgr2=0x7f66480fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 shutdown_connections 2024-04-22T06:37:47.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.630+0000 7f664e0ce640 1 -- 172.21.15.105:0/1856989731 wait complete. 2024-04-22T06:37:47.640 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:47.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 -- 172.21.15.105:0/3173831711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8102190 msgr2=0x7f62f8102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 --2- 172.21.15.105:0/3173831711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8102190 0x7f62f8102570 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009ca0 tx=0x7f62f402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 -- 172.21.15.105:0/3173831711 shutdown_connections 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 --2- 172.21.15.105:0/3173831711 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8107260 0x7f62f810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 --2- 172.21.15.105:0/3173831711 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f8102b40 0x7f62f8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 --2- 172.21.15.105:0/3173831711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8102190 0x7f62f8102570 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 -- 172.21.15.105:0/3173831711 >> 172.21.15.105:0/3173831711 conn(0x7f62f80793a0 msgr2=0x7f62f80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 -- 172.21.15.105:0/3173831711 shutdown_connections 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.741+0000 7f62ffe14640 1 -- 172.21.15.105:0/3173831711 wait complete. 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 Processor -- start 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 -- start start 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8102190 0x7f62f819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 0x7f62f8198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81122a0 con 0x7f62f8107260 2024-04-22T06:37:47.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62f8112120 con 0x7f62f81986a0 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62ffe14640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f62f8112420 con 0x7f62f8102190 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62fd3af640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62fd3af640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40844/0 (socket says 172.21.15.105:40844) 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.742+0000 7f62fd3af640 1 -- 172.21.15.105:0/2681968178 learned_addr learned my addr 172.21.15.105:0/2681968178 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fdbb0640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8102190 0x7f62f819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fe3b1640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 0x7f62f8198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 -- 172.21.15.105:0/2681968178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8102190 msgr2=0x7f62f819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8102190 0x7f62f819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 -- 172.21.15.105:0/2681968178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 msgr2=0x7f62f8198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 0x7f62f8198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 -- 172.21.15.105:0/2681968178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f81992a0 con 0x7f62f8107260 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fe3b1640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 0x7f62f8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62fd3af640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f62e800b450 tx=0x7f62e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8015020 con 0x7f62f8107260 2024-04-22T06:37:47.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62e8011d40 con 0x7f62f8107260 2024-04-22T06:37:47.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e800f340 con 0x7f62f8107260 2024-04-22T06:37:47.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f81a5380 con 0x7f62f8107260 2024-04-22T06:37:47.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.743+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f81a5780 con 0x7f62f8107260 2024-04-22T06:37:47.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.744+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f8103660 con 0x7f62f8107260 2024-04-22T06:37:47.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.745+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f62e8018020 con 0x7f62f8107260 2024-04-22T06:37:47.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.745+0000 7f62eeffd640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 0x7f62e40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:47.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.746+0000 7f62fdbb0640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 0x7f62e40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:47.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.746+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f62e80d0480 con 0x7f62f8107260 2024-04-22T06:37:47.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.746+0000 7f62fdbb0640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 0x7f62e40a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006010 tx=0x7f62f40099e0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.750+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62e801f020 con 0x7f62f8107260 2024-04-22T06:37:47.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.929+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f62f80691c0 con 0x7f62f8107260 2024-04-22T06:37:47.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.929+0000 7f62eeffd640 1 -- 172.21.15.105:0/2681968178 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f62e800d040 con 0x7f62f8107260 2024-04-22T06:37:47.929 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 msgr2=0x7f62e40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 0x7f62e40a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62f4006010 tx=0x7f62f40099e0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 msgr2=0x7f62f819eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f62e800b450 tx=0x7f62e800b910 comp rx=0 tx=0).stop 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 shutdown_connections 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f62e40a3e40 0x7f62e40a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62f81986a0 0x7f62f8198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62f8107260 0x7f62f819eb20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 --2- 172.21.15.105:0/2681968178 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62f8102190 0x7f62f819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:47.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 >> 172.21.15.105:0/2681968178 conn(0x7f62f80793a0 msgr2=0x7f62f8100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:47.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.931+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 shutdown_connections 2024-04-22T06:37:47.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:47.932+0000 7f62ffe14640 1 -- 172.21.15.105:0/2681968178 wait complete. 2024-04-22T06:37:47.942 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-22T06:37:47.942 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 -- 172.21.15.105:0/1875217322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 msgr2=0x7f4ef810ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 --2- 172.21.15.105:0/1875217322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef810ede0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009cd0 tx=0x7f4ef002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 -- 172.21.15.105:0/1875217322 shutdown_connections 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 --2- 172.21.15.105:0/1875217322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef810ede0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 --2- 172.21.15.105:0/1875217322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 0x7f4ef8100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 --2- 172.21.15.105:0/1875217322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 0x7f4ef8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 -- 172.21.15.105:0/1875217322 >> 172.21.15.105:0/1875217322 conn(0x7f4ef8077450 msgr2=0x7f4ef80fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 -- 172.21.15.105:0/1875217322 shutdown_connections 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.041+0000 7f4efea31640 1 -- 172.21.15.105:0/1875217322 wait complete. 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 Processor -- start 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 -- start start 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 0x7f4ef81016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 0x7f4ef8102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81122e0 con 0x7f4ef8100e80 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8112160 con 0x7f4ef8106f50 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4efea31640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef8112460 con 0x7f4ef81004c0 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4ef77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4ef77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40862/0 (socket says 172.21.15.105:40862) 2024-04-22T06:37:48.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.042+0000 7f4ef77fe640 1 -- 172.21.15.105:0/1410078439 learned_addr learned my addr 172.21.15.105:0/1410078439 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef7fff640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 0x7f4ef81016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4efcfce640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 0x7f4ef8102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 -- 172.21.15.105:0/1410078439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 msgr2=0x7f4ef81016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 0x7f4ef81016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 -- 172.21.15.105:0/1410078439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 msgr2=0x7f4ef8102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 0x7f4ef8102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 -- 172.21.15.105:0/1410078439 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef81a6cc0 con 0x7f4ef8100e80 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef77fe640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00d950 tx=0x7f4eec00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec017070 con 0x7f4ef8100e80 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eec00de80 con 0x7f4ef8100e80 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81a6fb0 con 0x7f4ef8100e80 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec00f620 con 0x7f4ef8100e80 2024-04-22T06:37:48.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.043+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef81a7470 con 0x7f4ef8100e80 2024-04-22T06:37:48.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.044+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef8069230 con 0x7f4ef8100e80 2024-04-22T06:37:48.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.045+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f4eec009c30 con 0x7f4ef8100e80 2024-04-22T06:37:48.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.046+0000 7f4ef57fa640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 0x7f4ee00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.046+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4eec0d0620 con 0x7f4ef8100e80 2024-04-22T06:37:48.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.046+0000 7f4ef7fff640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 0x7f4ee00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.046+0000 7f4ef7fff640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 0x7f4ee00a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009990 tx=0x7f4ee8009110 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.049+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4eec09cae0 con 0x7f4ef8100e80 2024-04-22T06:37:48.224 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.224+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ef8062390 con 0x7f4ee00a3e40 2024-04-22T06:37:48.231 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.231+0000 7f4ef57fa640 1 -- 172.21.15.105:0/1410078439 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257808 (secure 0 0 0) 0x7f4ef8062390 con 0x7f4ee00a3e40 2024-04-22T06:37:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:48.235 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 msgr2=0x7f4ee00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 0x7f4ee00a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009990 tx=0x7f4ee8009110 comp rx=0 tx=0).stop 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 msgr2=0x7f4ef8101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00d950 tx=0x7f4eec00dd50 comp rx=0 tx=0).stop 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 shutdown_connections 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ee00a3e40 0x7f4ee00a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef8106f50 0x7f4ef8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef8100e80 0x7f4ef8101be0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 --2- 172.21.15.105:0/1410078439 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef81004c0 0x7f4ef81016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 >> 172.21.15.105:0/1410078439 conn(0x7f4ef8077450 msgr2=0x7f4ef8104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 shutdown_connections 2024-04-22T06:37:48.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.235+0000 7f4efea31640 1 -- 172.21.15.105:0/1410078439 wait complete. 2024-04-22T06:37:48.244 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:48 smithi105 ceph-mon[27916]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-22T06:37:48.244 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2359681399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:48.244 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3171880173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.245 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1856989731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.245 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2681968178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.246 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:37:46.943698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193055,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1545,"num_read_kb":1102,"num_write":581,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389732,"kb_used_data":15600,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552092,"statfs":{"total":1151940427776,"available":1151541342208,"internally_reserved":0,"allocated":15974400,"data_stored":9390669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":382973243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4084,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":244,"num_write":31,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.987850"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975306+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975306+0000","last_peered":"2024-04-22T06:37:36.975306+0000","last_clean":"2024-04-22T06:37:36.975306+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975306+0000","last_undegraded":"2024-04-22T06:37:36.975306+0000","last_fullsized":"2024-04-22T06:37:36.975306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.150673+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:41.150673+0000","last_peered":"2024-04-22T06:37:41.150673+0000","last_clean":"2024-04-22T06:37:41.150673+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:41.150673+0000","last_undegraded":"2024-04-22T06:37:41.150673+0000","last_fullsized":"2024-04-22T06:37:41.150673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974851+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974851+0000","last_peered":"2024-04-22T06:37:36.974851+0000","last_clean":"2024-04-22T06:37:36.974851+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974851+0000","last_undegraded":"2024-04-22T06:37:36.974851+0000","last_fullsized":"2024-04-22T06:37:36.974851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":148,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151993+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:41.151993+0000","last_peered":"2024-04-22T06:37:41.151993+0000","last_clean":"2024-04-22T06:37:41.151993+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:41.151993+0000","last_undegraded":"2024-04-22T06:37:41.151993+0000","last_fullsized":"2024-04-22T06:37:41.151993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":261,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152651+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:42.152651+0000","last_peered":"2024-04-22T06:37:42.152651+0000","last_clean":"2024-04-22T06:37:42.152651+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:42.152651+0000","last_undegraded":"2024-04-22T06:37:42.152651+0000","last_fullsized":"2024-04-22T06:37:42.152651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":305,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.188547+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:42.188547+0000","last_peered":"2024-04-22T06:37:42.188547+0000","last_clean":"2024-04-22T06:37:42.188547+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:42.188547+0000","last_undegraded":"2024-04-22T06:37:42.188547+0000","last_fullsized":"2024-04-22T06:37:42.188547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67638,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":246,"num_read_kb":190,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'10","reported_seq":153,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151564+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:41.151564+0000","last_peered":"2024-04-22T06:37:41.151564+0000","last_clean":"2024-04-22T06:37:41.151564+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:41.151564+0000","last_undegraded":"2024-04-22T06:37:41.151564+0000","last_fullsized":"2024-04-22T06:37:41.151564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933504+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933504+0000","last_peered":"2024-04-22T06:37:36.933504+0000","last_clean":"2024-04-22T06:37:36.933504+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933504+0000","last_undegraded":"2024-04-22T06:37:36.933504+0000","last_fullsized":"2024-04-22T06:37:36.933504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151417+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:41.151417+0000","last_peered":"2024-04-22T06:37:41.151417+0000","last_clean":"2024-04-22T06:37:41.151417+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:41.151417+0000","last_undegraded":"2024-04-22T06:37:41.151417+0000","last_fullsized":"2024-04-22T06:37:41.151417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152403+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:42.152403+0000","last_peered":"2024-04-22T06:37:42.152403+0000","last_clean":"2024-04-22T06:37:42.152403+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:42.152403+0000","last_undegraded":"2024-04-22T06:37:42.152403+0000","last_fullsized":"2024-04-22T06:37:42.152403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"92'148","reported_seq":404,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:39.037085+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:39.037085+0000","last_peered":"2024-04-22T06:37:39.037085+0000","last_clean":"2024-04-22T06:37:39.037085+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:39.037085+0000","last_undegraded":"2024-04-22T06:37:39.037085+0000","last_fullsized":"2024-04-22T06:37:39.037085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975078+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975078+0000","last_peered":"2024-04-22T06:37:36.975078+0000","last_clean":"2024-04-22T06:37:36.975078+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975078+0000","last_undegraded":"2024-04-22T06:37:36.975078+0000","last_fullsized":"2024-04-22T06:37:36.975078+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'49","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.134795+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.134795+0000","last_peered":"2024-04-22T06:37:37.134795+0000","last_clean":"2024-04-22T06:37:37.134795+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.134795+0000","last_undegraded":"2024-04-22T06:37:37.134795+0000","last_fullsized":"2024-04-22T06:37:37.134795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974076+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974076+0000","last_peered":"2024-04-22T06:37:36.974076+0000","last_clean":"2024-04-22T06:37:36.974076+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974076+0000","last_undegraded":"2024-04-22T06:37:36.974076+0000","last_fullsized":"2024-04-22T06:37:36.974076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151888+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:41.151888+0000","last_peered":"2024-04-22T06:37:41.151888+0000","last_clean":"2024-04-22T06:37:41.151888+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:41.151888+0000","last_undegraded":"2024-04-22T06:37:41.151888+0000","last_fullsized":"2024-04-22T06:37:41.151888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602453,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1308,"num_read_kb":772,"num_write":378,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1807359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514538,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37148,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708004,"statfs":{"total":95995035648,"available":95956996096,"internally_reserved":0,"allocated":1855488,"data_stored":1277373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972404,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1081105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":554464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":555007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37072,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708080,"statfs":{"total":95995035648,"available":95957073920,"internally_reserved":0,"allocated":1777664,"data_stored":1209979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:48.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 -- 172.21.15.105:0/3207413594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c102b40 msgr2=0x7fe91c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 --2- 172.21.15.105:0/3207413594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c102b40 0x7fe91c106b80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe910009900 tx=0x7fe91002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 -- 172.21.15.105:0/3207413594 shutdown_connections 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 --2- 172.21.15.105:0/3207413594 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c107260 0x7fe91c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 --2- 172.21.15.105:0/3207413594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c102b40 0x7fe91c106b80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 --2- 172.21.15.105:0/3207413594 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c102190 0x7fe91c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 -- 172.21.15.105:0/3207413594 >> 172.21.15.105:0/3207413594 conn(0x7fe91c0793e0 msgr2=0x7fe91c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 -- 172.21.15.105:0/3207413594 shutdown_connections 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.385+0000 7fe92270d640 1 -- 172.21.15.105:0/3207413594 wait complete. 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 Processor -- start 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 -- start start 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 0x7fe91c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c19d020 0x7fe91c19d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c1121a0 con 0x7fe91c107260 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c112020 con 0x7fe91c102190 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe92270d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c112320 con 0x7fe91c19d020 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe91b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe91b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40882/0 (socket says 172.21.15.105:40882) 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe91b7fe640 1 -- 172.21.15.105:0/85546364 learned_addr learned my addr 172.21.15.105:0/85546364 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe91bfff640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 0x7fe91c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.386+0000 7fe920caa640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c19d020 0x7fe91c19d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 -- 172.21.15.105:0/85546364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c19d020 msgr2=0x7fe91c19d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c19d020 0x7fe91c19d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 -- 172.21.15.105:0/85546364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 msgr2=0x7fe91c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 0x7fe91c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 -- 172.21.15.105:0/85546364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c19db10 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91bfff640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 0x7fe91c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe91b7fe640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe910036040 tx=0x7fe9100045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910043020 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe91002cb70 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91c19dda0 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe910003bf0 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.387+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91c1a55c0 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.388+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91c069230 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.389+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fe910046020 con 0x7fe91c107260 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.390+0000 7fe9197fa640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 0x7fe9040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.390+0000 7fe91bfff640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 0x7fe9040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.390+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fe9100f2bb0 con 0x7fe91c107260 2024-04-22T06:37:48.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.390+0000 7fe91bfff640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 0x7fe9040a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c20 tx=0x7fe90c005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.394+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe91004d020 con 0x7fe91c107260 2024-04-22T06:37:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:48 smithi112 ceph-mon[31236]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-22T06:37:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2359681399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3171880173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1856989731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2681968178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:48 smithi177 ceph-mon[31346]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.1 KiB/s wr, 20 op/s 2024-04-22T06:37:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2359681399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3171880173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1856989731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2681968178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:48.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.567+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe91c10a4c0 con 0x7fe9040a3e40 2024-04-22T06:37:48.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.574+0000 7fe9197fa640 1 -- 172.21.15.105:0/85546364 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257808 (secure 0 0 0) 0x7fe91c10a4c0 con 0x7fe9040a3e40 2024-04-22T06:37:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:48.578 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 msgr2=0x7fe9040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 0x7fe9040a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c20 tx=0x7fe90c005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 msgr2=0x7fe91c198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe910036040 tx=0x7fe9100045e0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 shutdown_connections 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe9040a3e40 0x7fe9040a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe91c19d020 0x7fe91c19d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe91c107260 0x7fe91c198ca0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 --2- 172.21.15.105:0/85546364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe91c102190 0x7fe91c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.577+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 >> 172.21.15.105:0/85546364 conn(0x7fe91c0793e0 msgr2=0x7fe91c0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.578+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 shutdown_connections 2024-04-22T06:37:48.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.578+0000 7fe92270d640 1 -- 172.21.15.105:0/85546364 wait complete. 2024-04-22T06:37:48.588 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:37:46.943698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193055,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1545,"num_read_kb":1102,"num_write":581,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389732,"kb_used_data":15600,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552092,"statfs":{"total":1151940427776,"available":1151541342208,"internally_reserved":0,"allocated":15974400,"data_stored":9390669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":382973243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4084,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":244,"num_write":31,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.987850"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975306+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975306+0000","last_peered":"2024-04-22T06:37:36.975306+0000","last_clean":"2024-04-22T06:37:36.975306+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975306+0000","last_undegraded":"2024-04-22T06:37:36.975306+0000","last_fullsized":"2024-04-22T06:37:36.975306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.150673+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:41.150673+0000","last_peered":"2024-04-22T06:37:41.150673+0000","last_clean":"2024-04-22T06:37:41.150673+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:41.150673+0000","last_undegraded":"2024-04-22T06:37:41.150673+0000","last_fullsized":"2024-04-22T06:37:41.150673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974851+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974851+0000","last_peered":"2024-04-22T06:37:36.974851+0000","last_clean":"2024-04-22T06:37:36.974851+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974851+0000","last_undegraded":"2024-04-22T06:37:36.974851+0000","last_fullsized":"2024-04-22T06:37:36.974851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":148,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151993+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:41.151993+0000","last_peered":"2024-04-22T06:37:41.151993+0000","last_clean":"2024-04-22T06:37:41.151993+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:41.151993+0000","last_undegraded":"2024-04-22T06:37:41.151993+0000","last_fullsized":"2024-04-22T06:37:41.151993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":261,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152651+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:42.152651+0000","last_peered":"2024-04-22T06:37:42.152651+0000","last_clean":"2024-04-22T06:37:42.152651+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:42.152651+0000","last_undegraded":"2024-04-22T06:37:42.152651+0000","last_fullsized":"2024-04-22T06:37:42.152651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":305,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.188547+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:42.188547+0000","last_peered":"2024-04-22T06:37:42.188547+0000","last_clean":"2024-04-22T06:37:42.188547+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:42.188547+0000","last_undegraded":"2024-04-22T06:37:42.188547+0000","last_fullsized":"2024-04-22T06:37:42.188547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67638,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":246,"num_read_kb":190,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'10","reported_seq":153,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151564+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:41.151564+0000","last_peered":"2024-04-22T06:37:41.151564+0000","last_clean":"2024-04-22T06:37:41.151564+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:41.151564+0000","last_undegraded":"2024-04-22T06:37:41.151564+0000","last_fullsized":"2024-04-22T06:37:41.151564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933504+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933504+0000","last_peered":"2024-04-22T06:37:36.933504+0000","last_clean":"2024-04-22T06:37:36.933504+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933504+0000","last_undegraded":"2024-04-22T06:37:36.933504+0000","last_fullsized":"2024-04-22T06:37:36.933504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151417+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:41.151417+0000","last_peered":"2024-04-22T06:37:41.151417+0000","last_clean":"2024-04-22T06:37:41.151417+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:41.151417+0000","last_undegraded":"2024-04-22T06:37:41.151417+0000","last_fullsized":"2024-04-22T06:37:41.151417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152403+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:42.152403+0000","last_peered":"2024-04-22T06:37:42.152403+0000","last_clean":"2024-04-22T06:37:42.152403+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:42.152403+0000","last_undegraded":"2024-04-22T06:37:42.152403+0000","last_fullsized":"2024-04-22T06:37:42.152403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"92'148","reported_seq":404,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:39.037085+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:39.037085+0000","last_peered":"2024-04-22T06:37:39.037085+0000","last_clean":"2024-04-22T06:37:39.037085+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:39.037085+0000","last_undegraded":"2024-04-22T06:37:39.037085+0000","last_fullsized":"2024-04-22T06:37:39.037085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975078+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975078+0000","last_peered":"2024-04-22T06:37:36.975078+0000","last_clean":"2024-04-22T06:37:36.975078+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975078+0000","last_undegraded":"2024-04-22T06:37:36.975078+0000","last_fullsized":"2024-04-22T06:37:36.975078+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'49","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.134795+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.134795+0000","last_peered":"2024-04-22T06:37:37.134795+0000","last_clean":"2024-04-22T06:37:37.134795+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.134795+0000","last_undegraded":"2024-04-22T06:37:37.134795+0000","last_fullsized":"2024-04-22T06:37:37.134795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974076+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974076+0000","last_peered":"2024-04-22T06:37:36.974076+0000","last_clean":"2024-04-22T06:37:36.974076+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974076+0000","last_undegraded":"2024-04-22T06:37:36.974076+0000","last_fullsized":"2024-04-22T06:37:36.974076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151888+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:41.151888+0000","last_peered":"2024-04-22T06:37:41.151888+0000","last_clean":"2024-04-22T06:37:41.151888+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:41.151888+0000","last_undegraded":"2024-04-22T06:37:41.151888+0000","last_fullsized":"2024-04-22T06:37:41.151888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602453,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1308,"num_read_kb":772,"num_write":378,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1807359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514538,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37148,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708004,"statfs":{"total":95995035648,"available":95956996096,"internally_reserved":0,"allocated":1855488,"data_stored":1277373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972404,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1081105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":554464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":555007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37072,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708080,"statfs":{"total":95995035648,"available":95957073920,"internally_reserved":0,"allocated":1777664,"data_stored":1209979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:48.592 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-22T06:37:48.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:37:48.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.727+0000 7efd39e70640 1 -- 172.21.15.105:0/2322732587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 msgr2=0x7efd34106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.727+0000 7efd39e70640 1 --2- 172.21.15.105:0/2322732587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd34106210 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efd24009930 tx=0x7efd2402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.727+0000 7efd39e70640 1 -- 172.21.15.105:0/2322732587 shutdown_connections 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.727+0000 7efd39e70640 1 --2- 172.21.15.105:0/2322732587 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd3410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 --2- 172.21.15.105:0/2322732587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 --2- 172.21.15.105:0/2322732587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd34106210 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 -- 172.21.15.105:0/2322732587 >> 172.21.15.105:0/2322732587 conn(0x7efd34077e30 msgr2=0x7efd34078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 -- 172.21.15.105:0/2322732587 shutdown_connections 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 -- 172.21.15.105:0/2322732587 wait complete. 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 Processor -- start 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 -- start start 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.728+0000 7efd39e70640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd34198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efd34112240 con 0x7efd34105e30 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efd341120c0 con 0x7efd341011a0 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efd341123c0 con 0x7efd34101b60 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40900/0 (socket says 172.21.15.105:40900) 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 -- 172.21.15.105:0/942655599 learned_addr learned my addr 172.21.15.105:0/942655599 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd337fe640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd32ffd640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd34198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 -- 172.21.15.105:0/942655599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 msgr2=0x7efd34198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd34198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 -- 172.21.15.105:0/942655599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 msgr2=0x7efd34198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 -- 172.21.15.105:0/942655599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3419d730 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd32ffd640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd34198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd337fe640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd33fff640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00cad0 tx=0x7efd2c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00ae20 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd2c005020 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3419da20 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c00f6b0 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.729+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd341a52b0 con 0x7efd34105e30 2024-04-22T06:37:48.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.730+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd34102570 con 0x7efd34105e30 2024-04-22T06:37:48.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.731+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7efd2c0051d0 con 0x7efd34105e30 2024-04-22T06:37:48.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.732+0000 7efd30ff9640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 0x7efd1c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:48.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.732+0000 7efd337fe640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 0x7efd1c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:48.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.732+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7efd2c0d0b30 con 0x7efd34105e30 2024-04-22T06:37:48.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.732+0000 7efd337fe640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 0x7efd1c0a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd24036040 tx=0x7efd24000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:48.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.736+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd2c014030 con 0x7efd34105e30 2024-04-22T06:37:48.913 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.914+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd34062390 con 0x7efd1c0a3e40 2024-04-22T06:37:48.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.921+0000 7efd30ff9640 1 -- 172.21.15.105:0/942655599 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257808 (secure 0 0 0) 0x7efd34062390 con 0x7efd1c0a3e40 2024-04-22T06:37:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:48.924 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 msgr2=0x7efd1c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 0x7efd1c0a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efd24036040 tx=0x7efd24000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 msgr2=0x7efd3419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00cad0 tx=0x7efd2c00cf90 comp rx=0 tx=0).stop 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 shutdown_connections 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efd1c0a3e40 0x7efd1c0a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efd34105e30 0x7efd3419d000 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efd34101b60 0x7efd34198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 --2- 172.21.15.105:0/942655599 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd341011a0 0x7efd34198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:48.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 >> 172.21.15.105:0/942655599 conn(0x7efd34077e30 msgr2=0x7efd340fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:48.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 shutdown_connections 2024-04-22T06:37:48.927 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:48.924+0000 7efd39e70640 1 -- 172.21.15.105:0/942655599 wait complete. 2024-04-22T06:37:48.935 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-22T06:37:46.943698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1193055,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1545,"num_read_kb":1102,"num_write":581,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1180804,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389732,"kb_used_data":15600,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124552092,"statfs":{"total":1151940427776,"available":1151541342208,"internally_reserved":0,"allocated":15974400,"data_stored":9390669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":382973243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4084,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":157,"num_read_kb":244,"num_write":31,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.987850"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000277+0000","last_change":"2024-04-22T06:35:40.807025+0000","last_active":"2024-04-22T06:37:37.000277+0000","last_peered":"2024-04-22T06:37:37.000277+0000","last_clean":"2024-04-22T06:37:37.000277+0000","last_became_active":"2024-04-22T06:35:40.806595+0000","last_became_peered":"2024-04-22T06:35:40.806595+0000","last_unstale":"2024-04-22T06:37:37.000277+0000","last_undegraded":"2024-04-22T06:37:37.000277+0000","last_fullsized":"2024-04-22T06:37:37.000277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:58:23.140835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975356+0000","last_change":"2024-04-22T06:35:40.812202+0000","last_active":"2024-04-22T06:37:36.975356+0000","last_peered":"2024-04-22T06:37:36.975356+0000","last_clean":"2024-04-22T06:37:36.975356+0000","last_became_active":"2024-04-22T06:35:40.811544+0000","last_became_peered":"2024-04-22T06:35:40.811544+0000","last_unstale":"2024-04-22T06:37:36.975356+0000","last_undegraded":"2024-04-22T06:37:36.975356+0000","last_fullsized":"2024-04-22T06:37:36.975356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:20:01.022524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975327+0000","last_change":"2024-04-22T06:35:40.812053+0000","last_active":"2024-04-22T06:37:36.975327+0000","last_peered":"2024-04-22T06:37:36.975327+0000","last_clean":"2024-04-22T06:37:36.975327+0000","last_became_active":"2024-04-22T06:35:40.811598+0000","last_became_peered":"2024-04-22T06:35:40.811598+0000","last_unstale":"2024-04-22T06:37:36.975327+0000","last_undegraded":"2024-04-22T06:37:36.975327+0000","last_fullsized":"2024-04-22T06:37:36.975327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:49:02.889282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975306+0000","last_change":"2024-04-22T06:35:40.816660+0000","last_active":"2024-04-22T06:37:36.975306+0000","last_peered":"2024-04-22T06:37:36.975306+0000","last_clean":"2024-04-22T06:37:36.975306+0000","last_became_active":"2024-04-22T06:35:40.816316+0000","last_became_peered":"2024-04-22T06:35:40.816316+0000","last_unstale":"2024-04-22T06:37:36.975306+0000","last_undegraded":"2024-04-22T06:37:36.975306+0000","last_fullsized":"2024-04-22T06:37:36.975306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:26:16.322822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933086+0000","last_change":"2024-04-22T06:35:40.815637+0000","last_active":"2024-04-22T06:37:36.933086+0000","last_peered":"2024-04-22T06:37:36.933086+0000","last_clean":"2024-04-22T06:37:36.933086+0000","last_became_active":"2024-04-22T06:35:40.815310+0000","last_became_peered":"2024-04-22T06:35:40.815310+0000","last_unstale":"2024-04-22T06:37:36.933086+0000","last_undegraded":"2024-04-22T06:37:36.933086+0000","last_fullsized":"2024-04-22T06:37:36.933086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:38:42.012819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974738+0000","last_change":"2024-04-22T06:35:40.808942+0000","last_active":"2024-04-22T06:37:36.974738+0000","last_peered":"2024-04-22T06:37:36.974738+0000","last_clean":"2024-04-22T06:37:36.974738+0000","last_became_active":"2024-04-22T06:35:40.808587+0000","last_became_peered":"2024-04-22T06:35:40.808587+0000","last_unstale":"2024-04-22T06:37:36.974738+0000","last_undegraded":"2024-04-22T06:37:36.974738+0000","last_fullsized":"2024-04-22T06:37:36.974738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:50:09.529990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934211+0000","last_change":"2024-04-22T06:35:40.818640+0000","last_active":"2024-04-22T06:37:36.934211+0000","last_peered":"2024-04-22T06:37:36.934211+0000","last_clean":"2024-04-22T06:37:36.934211+0000","last_became_active":"2024-04-22T06:35:40.818143+0000","last_became_peered":"2024-04-22T06:35:40.818143+0000","last_unstale":"2024-04-22T06:37:36.934211+0000","last_undegraded":"2024-04-22T06:37:36.934211+0000","last_fullsized":"2024-04-22T06:37:36.934211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:16:09.816150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973938+0000","last_change":"2024-04-22T06:35:40.818156+0000","last_active":"2024-04-22T06:37:36.973938+0000","last_peered":"2024-04-22T06:37:36.973938+0000","last_clean":"2024-04-22T06:37:36.973938+0000","last_became_active":"2024-04-22T06:35:40.817839+0000","last_became_peered":"2024-04-22T06:35:40.817839+0000","last_unstale":"2024-04-22T06:37:36.973938+0000","last_undegraded":"2024-04-22T06:37:36.973938+0000","last_fullsized":"2024-04-22T06:37:36.973938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:13:48.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975916+0000","last_change":"2024-04-22T06:35:40.815924+0000","last_active":"2024-04-22T06:37:36.975916+0000","last_peered":"2024-04-22T06:37:36.975916+0000","last_clean":"2024-04-22T06:37:36.975916+0000","last_became_active":"2024-04-22T06:35:40.815564+0000","last_became_peered":"2024-04-22T06:35:40.815564+0000","last_unstale":"2024-04-22T06:37:36.975916+0000","last_undegraded":"2024-04-22T06:37:36.975916+0000","last_fullsized":"2024-04-22T06:37:36.975916+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:59.997001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975461+0000","last_change":"2024-04-22T06:35:40.808973+0000","last_active":"2024-04-22T06:37:36.975461+0000","last_peered":"2024-04-22T06:37:36.975461+0000","last_clean":"2024-04-22T06:37:36.975461+0000","last_became_active":"2024-04-22T06:35:40.808609+0000","last_became_peered":"2024-04-22T06:35:40.808609+0000","last_unstale":"2024-04-22T06:37:36.975461+0000","last_undegraded":"2024-04-22T06:37:36.975461+0000","last_fullsized":"2024-04-22T06:37:36.975461+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:53:37.150947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934520+0000","last_change":"2024-04-22T06:35:40.815129+0000","last_active":"2024-04-22T06:37:36.934520+0000","last_peered":"2024-04-22T06:37:36.934520+0000","last_clean":"2024-04-22T06:37:36.934520+0000","last_became_active":"2024-04-22T06:35:40.814779+0000","last_became_peered":"2024-04-22T06:35:40.814779+0000","last_unstale":"2024-04-22T06:37:36.934520+0000","last_undegraded":"2024-04-22T06:37:36.934520+0000","last_fullsized":"2024-04-22T06:37:36.934520+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:30:45.853994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929535+0000","last_change":"2024-04-22T06:36:45.787779+0000","last_active":"2024-04-22T06:37:36.929535+0000","last_peered":"2024-04-22T06:37:36.929535+0000","last_clean":"2024-04-22T06:37:36.929535+0000","last_became_active":"2024-04-22T06:36:45.787376+0000","last_became_peered":"2024-04-22T06:36:45.787376+0000","last_unstale":"2024-04-22T06:37:36.929535+0000","last_undegraded":"2024-04-22T06:37:36.929535+0000","last_fullsized":"2024-04-22T06:37:36.929535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:00:00.972811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974516+0000","last_change":"2024-04-22T06:35:40.815730+0000","last_active":"2024-04-22T06:37:36.974516+0000","last_peered":"2024-04-22T06:37:36.974516+0000","last_clean":"2024-04-22T06:37:36.974516+0000","last_became_active":"2024-04-22T06:35:40.815367+0000","last_became_peered":"2024-04-22T06:35:40.815367+0000","last_unstale":"2024-04-22T06:37:36.974516+0000","last_undegraded":"2024-04-22T06:37:36.974516+0000","last_fullsized":"2024-04-22T06:37:36.974516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:21:14.250910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974717+0000","last_change":"2024-04-22T06:35:40.809583+0000","last_active":"2024-04-22T06:37:36.974717+0000","last_peered":"2024-04-22T06:37:36.974717+0000","last_clean":"2024-04-22T06:37:36.974717+0000","last_became_active":"2024-04-22T06:35:40.809152+0000","last_became_peered":"2024-04-22T06:35:40.809152+0000","last_unstale":"2024-04-22T06:37:36.974717+0000","last_undegraded":"2024-04-22T06:37:36.974717+0000","last_fullsized":"2024-04-22T06:37:36.974717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:19:42.471732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974393+0000","last_change":"2024-04-22T06:35:40.812546+0000","last_active":"2024-04-22T06:37:36.974393+0000","last_peered":"2024-04-22T06:37:36.974393+0000","last_clean":"2024-04-22T06:37:36.974393+0000","last_became_active":"2024-04-22T06:35:40.812225+0000","last_became_peered":"2024-04-22T06:35:40.812225+0000","last_unstale":"2024-04-22T06:37:36.974393+0000","last_undegraded":"2024-04-22T06:37:36.974393+0000","last_fullsized":"2024-04-22T06:37:36.974393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:13:43.366674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975452+0000","last_change":"2024-04-22T06:35:40.816868+0000","last_active":"2024-04-22T06:37:36.975452+0000","last_peered":"2024-04-22T06:37:36.975452+0000","last_clean":"2024-04-22T06:37:36.975452+0000","last_became_active":"2024-04-22T06:35:40.816494+0000","last_became_peered":"2024-04-22T06:35:40.816494+0000","last_unstale":"2024-04-22T06:37:36.975452+0000","last_undegraded":"2024-04-22T06:37:36.975452+0000","last_fullsized":"2024-04-22T06:37:36.975452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:08:56.912568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000409+0000","last_change":"2024-04-22T06:35:40.806235+0000","last_active":"2024-04-22T06:37:37.000409+0000","last_peered":"2024-04-22T06:37:37.000409+0000","last_clean":"2024-04-22T06:37:37.000409+0000","last_became_active":"2024-04-22T06:35:40.805894+0000","last_became_peered":"2024-04-22T06:35:40.805894+0000","last_unstale":"2024-04-22T06:37:37.000409+0000","last_undegraded":"2024-04-22T06:37:37.000409+0000","last_fullsized":"2024-04-22T06:37:37.000409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:12:52.448988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929595+0000","last_change":"2024-04-22T06:35:40.808869+0000","last_active":"2024-04-22T06:37:36.929595+0000","last_peered":"2024-04-22T06:37:36.929595+0000","last_clean":"2024-04-22T06:37:36.929595+0000","last_became_active":"2024-04-22T06:35:40.808511+0000","last_became_peered":"2024-04-22T06:35:40.808511+0000","last_unstale":"2024-04-22T06:37:36.929595+0000","last_undegraded":"2024-04-22T06:37:36.929595+0000","last_fullsized":"2024-04-22T06:37:36.929595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:41:25.207440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974293+0000","last_change":"2024-04-22T06:35:40.820026+0000","last_active":"2024-04-22T06:37:36.974293+0000","last_peered":"2024-04-22T06:37:36.974293+0000","last_clean":"2024-04-22T06:37:36.974293+0000","last_became_active":"2024-04-22T06:35:40.819648+0000","last_became_peered":"2024-04-22T06:35:40.819648+0000","last_unstale":"2024-04-22T06:37:36.974293+0000","last_undegraded":"2024-04-22T06:37:36.974293+0000","last_fullsized":"2024-04-22T06:37:36.974293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:06:27.004346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'14","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.150673+0000","last_change":"2024-04-22T06:35:39.803422+0000","last_active":"2024-04-22T06:37:41.150673+0000","last_peered":"2024-04-22T06:37:41.150673+0000","last_clean":"2024-04-22T06:37:41.150673+0000","last_became_active":"2024-04-22T06:35:39.803131+0000","last_became_peered":"2024-04-22T06:35:39.803131+0000","last_unstale":"2024-04-22T06:37:41.150673+0000","last_undegraded":"2024-04-22T06:37:41.150673+0000","last_fullsized":"2024-04-22T06:37:41.150673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:56:28.397769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974568+0000","last_change":"2024-04-22T06:35:40.819780+0000","last_active":"2024-04-22T06:37:36.974568+0000","last_peered":"2024-04-22T06:37:36.974568+0000","last_clean":"2024-04-22T06:37:36.974568+0000","last_became_active":"2024-04-22T06:35:40.819285+0000","last_became_peered":"2024-04-22T06:35:40.819285+0000","last_unstale":"2024-04-22T06:37:36.974568+0000","last_undegraded":"2024-04-22T06:37:36.974568+0000","last_fullsized":"2024-04-22T06:37:36.974568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:38:10.936949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974851+0000","last_change":"2024-04-22T06:35:39.811101+0000","last_active":"2024-04-22T06:37:36.974851+0000","last_peered":"2024-04-22T06:37:36.974851+0000","last_clean":"2024-04-22T06:37:36.974851+0000","last_became_active":"2024-04-22T06:35:39.810761+0000","last_became_peered":"2024-04-22T06:35:39.810761+0000","last_unstale":"2024-04-22T06:37:36.974851+0000","last_undegraded":"2024-04-22T06:37:36.974851+0000","last_fullsized":"2024-04-22T06:37:36.974851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:49:14.538785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975561+0000","last_change":"2024-04-22T06:35:40.805605+0000","last_active":"2024-04-22T06:37:36.975561+0000","last_peered":"2024-04-22T06:37:36.975561+0000","last_clean":"2024-04-22T06:37:36.975561+0000","last_became_active":"2024-04-22T06:35:40.805232+0000","last_became_peered":"2024-04-22T06:35:40.805232+0000","last_unstale":"2024-04-22T06:37:36.975561+0000","last_undegraded":"2024-04-22T06:37:36.975561+0000","last_fullsized":"2024-04-22T06:37:36.975561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:43:39.821999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000106+0000","last_change":"2024-04-22T06:36:45.798607+0000","last_active":"2024-04-22T06:37:37.000106+0000","last_peered":"2024-04-22T06:37:37.000106+0000","last_clean":"2024-04-22T06:37:37.000106+0000","last_became_active":"2024-04-22T06:36:45.786682+0000","last_became_peered":"2024-04-22T06:36:45.786682+0000","last_unstale":"2024-04-22T06:37:37.000106+0000","last_undegraded":"2024-04-22T06:37:37.000106+0000","last_fullsized":"2024-04-22T06:37:37.000106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:07:02.050453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975638+0000","last_change":"2024-04-22T06:35:40.819643+0000","last_active":"2024-04-22T06:37:36.975638+0000","last_peered":"2024-04-22T06:37:36.975638+0000","last_clean":"2024-04-22T06:37:36.975638+0000","last_became_active":"2024-04-22T06:35:40.819304+0000","last_became_peered":"2024-04-22T06:35:40.819304+0000","last_unstale":"2024-04-22T06:37:36.975638+0000","last_undegraded":"2024-04-22T06:37:36.975638+0000","last_fullsized":"2024-04-22T06:37:36.975638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:08:27.626293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933452+0000","last_change":"2024-04-22T06:35:39.800637+0000","last_active":"2024-04-22T06:37:36.933452+0000","last_peered":"2024-04-22T06:37:36.933452+0000","last_clean":"2024-04-22T06:37:36.933452+0000","last_became_active":"2024-04-22T06:35:39.800171+0000","last_became_peered":"2024-04-22T06:35:39.800171+0000","last_unstale":"2024-04-22T06:37:36.933452+0000","last_undegraded":"2024-04-22T06:37:36.933452+0000","last_fullsized":"2024-04-22T06:37:36.933452+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:35.574990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974706+0000","last_change":"2024-04-22T06:35:40.821316+0000","last_active":"2024-04-22T06:37:36.974706+0000","last_peered":"2024-04-22T06:37:36.974706+0000","last_clean":"2024-04-22T06:37:36.974706+0000","last_became_active":"2024-04-22T06:35:40.820908+0000","last_became_peered":"2024-04-22T06:35:40.820908+0000","last_unstale":"2024-04-22T06:37:36.974706+0000","last_undegraded":"2024-04-22T06:37:36.974706+0000","last_fullsized":"2024-04-22T06:37:36.974706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:49:16.285690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":148,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151993+0000","last_change":"2024-04-22T06:35:39.804367+0000","last_active":"2024-04-22T06:37:41.151993+0000","last_peered":"2024-04-22T06:37:41.151993+0000","last_clean":"2024-04-22T06:37:41.151993+0000","last_became_active":"2024-04-22T06:35:39.803987+0000","last_became_peered":"2024-04-22T06:35:39.803987+0000","last_unstale":"2024-04-22T06:37:41.151993+0000","last_undegraded":"2024-04-22T06:37:41.151993+0000","last_fullsized":"2024-04-22T06:37:41.151993+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:29:40.129505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932974+0000","last_change":"2024-04-22T06:35:40.814883+0000","last_active":"2024-04-22T06:37:36.932974+0000","last_peered":"2024-04-22T06:37:36.932974+0000","last_clean":"2024-04-22T06:37:36.932974+0000","last_became_active":"2024-04-22T06:35:40.814536+0000","last_became_peered":"2024-04-22T06:35:40.814536+0000","last_unstale":"2024-04-22T06:37:36.932974+0000","last_undegraded":"2024-04-22T06:37:36.932974+0000","last_fullsized":"2024-04-22T06:37:36.932974+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:29:39.952712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974513+0000","last_change":"2024-04-22T06:35:39.807291+0000","last_active":"2024-04-22T06:37:36.974513+0000","last_peered":"2024-04-22T06:37:36.974513+0000","last_clean":"2024-04-22T06:37:36.974513+0000","last_became_active":"2024-04-22T06:35:39.807055+0000","last_became_peered":"2024-04-22T06:35:39.807055+0000","last_unstale":"2024-04-22T06:37:36.974513+0000","last_undegraded":"2024-04-22T06:37:36.974513+0000","last_fullsized":"2024-04-22T06:37:36.974513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:04:57.444245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974676+0000","last_change":"2024-04-22T06:35:40.818593+0000","last_active":"2024-04-22T06:37:36.974676+0000","last_peered":"2024-04-22T06:37:36.974676+0000","last_clean":"2024-04-22T06:37:36.974676+0000","last_became_active":"2024-04-22T06:35:40.818195+0000","last_became_peered":"2024-04-22T06:35:40.818195+0000","last_unstale":"2024-04-22T06:37:36.974676+0000","last_undegraded":"2024-04-22T06:37:36.974676+0000","last_fullsized":"2024-04-22T06:37:36.974676+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:25:22.512706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975207+0000","last_change":"2024-04-22T06:35:39.801910+0000","last_active":"2024-04-22T06:37:36.975207+0000","last_peered":"2024-04-22T06:37:36.975207+0000","last_clean":"2024-04-22T06:37:36.975207+0000","last_became_active":"2024-04-22T06:35:39.801555+0000","last_became_peered":"2024-04-22T06:35:39.801555+0000","last_unstale":"2024-04-22T06:37:36.975207+0000","last_undegraded":"2024-04-22T06:37:36.975207+0000","last_fullsized":"2024-04-22T06:37:36.975207+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:03:07.203959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974107+0000","last_change":"2024-04-22T06:35:40.808265+0000","last_active":"2024-04-22T06:37:36.974107+0000","last_peered":"2024-04-22T06:37:36.974107+0000","last_clean":"2024-04-22T06:37:36.974107+0000","last_became_active":"2024-04-22T06:35:40.807925+0000","last_became_peered":"2024-04-22T06:35:40.807925+0000","last_unstale":"2024-04-22T06:37:36.974107+0000","last_undegraded":"2024-04-22T06:37:36.974107+0000","last_fullsized":"2024-04-22T06:37:36.974107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:41:09.488165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974034+0000","last_change":"2024-04-22T06:35:39.802231+0000","last_active":"2024-04-22T06:37:36.974034+0000","last_peered":"2024-04-22T06:37:36.974034+0000","last_clean":"2024-04-22T06:37:36.974034+0000","last_became_active":"2024-04-22T06:35:39.801901+0000","last_became_peered":"2024-04-22T06:35:39.801901+0000","last_unstale":"2024-04-22T06:37:36.974034+0000","last_undegraded":"2024-04-22T06:37:36.974034+0000","last_fullsized":"2024-04-22T06:37:36.974034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:41:43.092163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929657+0000","last_change":"2024-04-22T06:35:40.806112+0000","last_active":"2024-04-22T06:37:36.929657+0000","last_peered":"2024-04-22T06:37:36.929657+0000","last_clean":"2024-04-22T06:37:36.929657+0000","last_became_active":"2024-04-22T06:35:40.805764+0000","last_became_peered":"2024-04-22T06:35:40.805764+0000","last_unstale":"2024-04-22T06:37:36.929657+0000","last_undegraded":"2024-04-22T06:37:36.929657+0000","last_fullsized":"2024-04-22T06:37:36.929657+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:47:40.355018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'6","reported_seq":261,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152651+0000","last_change":"2024-04-22T06:35:39.809986+0000","last_active":"2024-04-22T06:37:42.152651+0000","last_peered":"2024-04-22T06:37:42.152651+0000","last_clean":"2024-04-22T06:37:42.152651+0000","last_became_active":"2024-04-22T06:35:39.809634+0000","last_became_peered":"2024-04-22T06:35:39.809634+0000","last_unstale":"2024-04-22T06:37:42.152651+0000","last_undegraded":"2024-04-22T06:37:42.152651+0000","last_fullsized":"2024-04-22T06:37:42.152651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:02:00.594995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975846+0000","last_change":"2024-04-22T06:35:40.821863+0000","last_active":"2024-04-22T06:37:36.975846+0000","last_peered":"2024-04-22T06:37:36.975846+0000","last_clean":"2024-04-22T06:37:36.975846+0000","last_became_active":"2024-04-22T06:35:40.821572+0000","last_became_peered":"2024-04-22T06:35:40.821572+0000","last_unstale":"2024-04-22T06:37:36.975846+0000","last_undegraded":"2024-04-22T06:37:36.975846+0000","last_fullsized":"2024-04-22T06:37:36.975846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:11:15.756134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934622+0000","last_change":"2024-04-22T06:35:39.802230+0000","last_active":"2024-04-22T06:37:36.934622+0000","last_peered":"2024-04-22T06:37:36.934622+0000","last_clean":"2024-04-22T06:37:36.934622+0000","last_became_active":"2024-04-22T06:35:39.801904+0000","last_became_peered":"2024-04-22T06:35:39.801904+0000","last_unstale":"2024-04-22T06:37:36.934622+0000","last_undegraded":"2024-04-22T06:37:36.934622+0000","last_fullsized":"2024-04-22T06:37:36.934622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:48:39.841910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975496+0000","last_change":"2024-04-22T06:35:40.814835+0000","last_active":"2024-04-22T06:37:36.975496+0000","last_peered":"2024-04-22T06:37:36.975496+0000","last_clean":"2024-04-22T06:37:36.975496+0000","last_became_active":"2024-04-22T06:35:40.814506+0000","last_became_peered":"2024-04-22T06:35:40.814506+0000","last_unstale":"2024-04-22T06:37:36.975496+0000","last_undegraded":"2024-04-22T06:37:36.975496+0000","last_fullsized":"2024-04-22T06:37:36.975496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:59:49.704806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974736+0000","last_change":"2024-04-22T06:35:39.807712+0000","last_active":"2024-04-22T06:37:36.974736+0000","last_peered":"2024-04-22T06:37:36.974736+0000","last_clean":"2024-04-22T06:37:36.974736+0000","last_became_active":"2024-04-22T06:35:39.806384+0000","last_became_peered":"2024-04-22T06:35:39.806384+0000","last_unstale":"2024-04-22T06:37:36.974736+0000","last_undegraded":"2024-04-22T06:37:36.974736+0000","last_fullsized":"2024-04-22T06:37:36.974736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:52:22.197683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975740+0000","last_change":"2024-04-22T06:36:45.787055+0000","last_active":"2024-04-22T06:37:36.975740+0000","last_peered":"2024-04-22T06:37:36.975740+0000","last_clean":"2024-04-22T06:37:36.975740+0000","last_became_active":"2024-04-22T06:36:45.786576+0000","last_became_peered":"2024-04-22T06:36:45.786576+0000","last_unstale":"2024-04-22T06:37:36.975740+0000","last_undegraded":"2024-04-22T06:37:36.975740+0000","last_fullsized":"2024-04-22T06:37:36.975740+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:40:50.291546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":305,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.188547+0000","last_change":"2024-04-22T06:35:39.805021+0000","last_active":"2024-04-22T06:37:42.188547+0000","last_peered":"2024-04-22T06:37:42.188547+0000","last_clean":"2024-04-22T06:37:42.188547+0000","last_became_active":"2024-04-22T06:35:39.804614+0000","last_became_peered":"2024-04-22T06:35:39.804614+0000","last_unstale":"2024-04-22T06:37:42.188547+0000","last_undegraded":"2024-04-22T06:37:42.188547+0000","last_fullsized":"2024-04-22T06:37:42.188547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:55:15.859661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67638,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":246,"num_read_kb":190,"num_write":40,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933329+0000","last_change":"2024-04-22T06:35:40.828263+0000","last_active":"2024-04-22T06:37:36.933329+0000","last_peered":"2024-04-22T06:37:36.933329+0000","last_clean":"2024-04-22T06:37:36.933329+0000","last_became_active":"2024-04-22T06:35:40.827867+0000","last_became_peered":"2024-04-22T06:35:40.827867+0000","last_unstale":"2024-04-22T06:37:36.933329+0000","last_undegraded":"2024-04-22T06:37:36.933329+0000","last_fullsized":"2024-04-22T06:37:36.933329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:13:00.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999919+0000","last_change":"2024-04-22T06:36:45.808159+0000","last_active":"2024-04-22T06:37:36.999919+0000","last_peered":"2024-04-22T06:37:36.999919+0000","last_clean":"2024-04-22T06:37:36.999919+0000","last_became_active":"2024-04-22T06:36:45.787539+0000","last_became_peered":"2024-04-22T06:36:45.787539+0000","last_unstale":"2024-04-22T06:37:36.999919+0000","last_undegraded":"2024-04-22T06:37:36.999919+0000","last_fullsized":"2024-04-22T06:37:36.999919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:22:28.290638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974172+0000","last_change":"2024-04-22T06:35:40.824528+0000","last_active":"2024-04-22T06:37:36.974172+0000","last_peered":"2024-04-22T06:37:36.974172+0000","last_clean":"2024-04-22T06:37:36.974172+0000","last_became_active":"2024-04-22T06:35:40.824134+0000","last_became_peered":"2024-04-22T06:35:40.824134+0000","last_unstale":"2024-04-22T06:37:36.974172+0000","last_undegraded":"2024-04-22T06:37:36.974172+0000","last_fullsized":"2024-04-22T06:37:36.974172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:28:05.230670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974486+0000","last_change":"2024-04-22T06:35:39.807416+0000","last_active":"2024-04-22T06:37:36.974486+0000","last_peered":"2024-04-22T06:37:36.974486+0000","last_clean":"2024-04-22T06:37:36.974486+0000","last_became_active":"2024-04-22T06:35:39.807041+0000","last_became_peered":"2024-04-22T06:35:39.807041+0000","last_unstale":"2024-04-22T06:37:36.974486+0000","last_undegraded":"2024-04-22T06:37:36.974486+0000","last_fullsized":"2024-04-22T06:37:36.974486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:48:44.787903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934438+0000","last_change":"2024-04-22T06:35:40.820225+0000","last_active":"2024-04-22T06:37:36.934438+0000","last_peered":"2024-04-22T06:37:36.934438+0000","last_clean":"2024-04-22T06:37:36.934438+0000","last_became_active":"2024-04-22T06:35:40.819012+0000","last_became_peered":"2024-04-22T06:35:40.819012+0000","last_unstale":"2024-04-22T06:37:36.934438+0000","last_undegraded":"2024-04-22T06:37:36.934438+0000","last_fullsized":"2024-04-22T06:37:36.934438+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:11:27.674181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'10","reported_seq":153,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151564+0000","last_change":"2024-04-22T06:35:39.799304+0000","last_active":"2024-04-22T06:37:41.151564+0000","last_peered":"2024-04-22T06:37:41.151564+0000","last_clean":"2024-04-22T06:37:41.151564+0000","last_became_active":"2024-04-22T06:35:39.798816+0000","last_became_peered":"2024-04-22T06:35:39.798816+0000","last_unstale":"2024-04-22T06:37:41.151564+0000","last_undegraded":"2024-04-22T06:37:41.151564+0000","last_fullsized":"2024-04-22T06:37:41.151564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:38.048774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2210,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934063+0000","last_change":"2024-04-22T06:35:40.820484+0000","last_active":"2024-04-22T06:37:36.934063+0000","last_peered":"2024-04-22T06:37:36.934063+0000","last_clean":"2024-04-22T06:37:36.934063+0000","last_became_active":"2024-04-22T06:35:40.820150+0000","last_became_peered":"2024-04-22T06:35:40.820150+0000","last_unstale":"2024-04-22T06:37:36.934063+0000","last_undegraded":"2024-04-22T06:37:36.934063+0000","last_fullsized":"2024-04-22T06:37:36.934063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:17:06.664484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:39.804746+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:39.804430+0000","last_became_peered":"2024-04-22T06:35:39.804430+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:31:38.566275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929461+0000","last_change":"2024-04-22T06:36:46.033411+0000","last_active":"2024-04-22T06:37:36.929461+0000","last_peered":"2024-04-22T06:37:36.929461+0000","last_clean":"2024-04-22T06:37:36.929461+0000","last_became_active":"2024-04-22T06:36:46.033049+0000","last_became_peered":"2024-04-22T06:36:46.033049+0000","last_unstale":"2024-04-22T06:37:36.929461+0000","last_undegraded":"2024-04-22T06:37:36.929461+0000","last_fullsized":"2024-04-22T06:37:36.929461+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:50:17.785540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"92'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933504+0000","last_change":"2024-04-22T06:35:39.797847+0000","last_active":"2024-04-22T06:37:36.933504+0000","last_peered":"2024-04-22T06:37:36.933504+0000","last_clean":"2024-04-22T06:37:36.933504+0000","last_became_active":"2024-04-22T06:35:39.797528+0000","last_became_peered":"2024-04-22T06:35:39.797528+0000","last_unstale":"2024-04-22T06:37:36.933504+0000","last_undegraded":"2024-04-22T06:37:36.933504+0000","last_fullsized":"2024-04-22T06:37:36.933504+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:45:35.491428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000175+0000","last_change":"2024-04-22T06:35:40.819095+0000","last_active":"2024-04-22T06:37:37.000175+0000","last_peered":"2024-04-22T06:37:37.000175+0000","last_clean":"2024-04-22T06:37:37.000175+0000","last_became_active":"2024-04-22T06:35:40.817690+0000","last_became_peered":"2024-04-22T06:35:40.817690+0000","last_unstale":"2024-04-22T06:37:37.000175+0000","last_undegraded":"2024-04-22T06:37:37.000175+0000","last_fullsized":"2024-04-22T06:37:37.000175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:59:37.899406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.929543+0000","last_change":"2024-04-22T06:35:39.808946+0000","last_active":"2024-04-22T06:37:36.929543+0000","last_peered":"2024-04-22T06:37:36.929543+0000","last_clean":"2024-04-22T06:37:36.929543+0000","last_became_active":"2024-04-22T06:35:39.808630+0000","last_became_peered":"2024-04-22T06:35:39.808630+0000","last_unstale":"2024-04-22T06:37:36.929543+0000","last_undegraded":"2024-04-22T06:37:36.929543+0000","last_fullsized":"2024-04-22T06:37:36.929543+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:35:55.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975082+0000","last_change":"2024-04-22T06:35:40.816405+0000","last_active":"2024-04-22T06:37:36.975082+0000","last_peered":"2024-04-22T06:37:36.975082+0000","last_clean":"2024-04-22T06:37:36.975082+0000","last_became_active":"2024-04-22T06:35:40.816065+0000","last_became_peered":"2024-04-22T06:35:40.816065+0000","last_unstale":"2024-04-22T06:37:36.975082+0000","last_undegraded":"2024-04-22T06:37:36.975082+0000","last_fullsized":"2024-04-22T06:37:36.975082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:57:56.375854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'7","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151417+0000","last_change":"2024-04-22T06:35:39.801078+0000","last_active":"2024-04-22T06:37:41.151417+0000","last_peered":"2024-04-22T06:37:41.151417+0000","last_clean":"2024-04-22T06:37:41.151417+0000","last_became_active":"2024-04-22T06:35:39.800759+0000","last_became_peered":"2024-04-22T06:35:39.800759+0000","last_unstale":"2024-04-22T06:37:41.151417+0000","last_undegraded":"2024-04-22T06:37:41.151417+0000","last_fullsized":"2024-04-22T06:37:41.151417+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:45:39.604454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974590+0000","last_change":"2024-04-22T06:35:40.807925+0000","last_active":"2024-04-22T06:37:36.974590+0000","last_peered":"2024-04-22T06:37:36.974590+0000","last_clean":"2024-04-22T06:37:36.974590+0000","last_became_active":"2024-04-22T06:35:40.807558+0000","last_became_peered":"2024-04-22T06:35:40.807558+0000","last_unstale":"2024-04-22T06:37:36.974590+0000","last_undegraded":"2024-04-22T06:37:36.974590+0000","last_fullsized":"2024-04-22T06:37:36.974590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:10:02.341562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974404+0000","last_change":"2024-04-22T06:35:40.814882+0000","last_active":"2024-04-22T06:37:36.974404+0000","last_peered":"2024-04-22T06:37:36.974404+0000","last_clean":"2024-04-22T06:37:36.974404+0000","last_became_active":"2024-04-22T06:35:40.813548+0000","last_became_peered":"2024-04-22T06:35:40.813548+0000","last_unstale":"2024-04-22T06:37:36.974404+0000","last_undegraded":"2024-04-22T06:37:36.974404+0000","last_fullsized":"2024-04-22T06:37:36.974404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:43:34.944594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:42.152403+0000","last_change":"2024-04-22T06:35:39.806261+0000","last_active":"2024-04-22T06:37:42.152403+0000","last_peered":"2024-04-22T06:37:42.152403+0000","last_clean":"2024-04-22T06:37:42.152403+0000","last_became_active":"2024-04-22T06:35:39.805852+0000","last_became_peered":"2024-04-22T06:35:39.805852+0000","last_unstale":"2024-04-22T06:37:42.152403+0000","last_undegraded":"2024-04-22T06:37:42.152403+0000","last_fullsized":"2024-04-22T06:37:42.152403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:29:45.806330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974358+0000","last_change":"2024-04-22T06:35:40.816711+0000","last_active":"2024-04-22T06:37:36.974358+0000","last_peered":"2024-04-22T06:37:36.974358+0000","last_clean":"2024-04-22T06:37:36.974358+0000","last_became_active":"2024-04-22T06:35:40.816380+0000","last_became_peered":"2024-04-22T06:35:40.816380+0000","last_unstale":"2024-04-22T06:37:36.974358+0000","last_undegraded":"2024-04-22T06:37:36.974358+0000","last_fullsized":"2024-04-22T06:37:36.974358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:19:36.318514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974655+0000","last_change":"2024-04-22T06:35:39.802212+0000","last_active":"2024-04-22T06:37:36.974655+0000","last_peered":"2024-04-22T06:37:36.974655+0000","last_clean":"2024-04-22T06:37:36.974655+0000","last_became_active":"2024-04-22T06:35:39.800676+0000","last_became_peered":"2024-04-22T06:35:39.800676+0000","last_unstale":"2024-04-22T06:37:36.974655+0000","last_undegraded":"2024-04-22T06:37:36.974655+0000","last_fullsized":"2024-04-22T06:37:36.974655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:11:25.823134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"92'148","reported_seq":404,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:39.037085+0000","last_change":"2024-04-22T06:34:31.012582+0000","last_active":"2024-04-22T06:37:39.037085+0000","last_peered":"2024-04-22T06:37:39.037085+0000","last_clean":"2024-04-22T06:37:39.037085+0000","last_became_active":"2024-04-22T06:34:31.012055+0000","last_became_peered":"2024-04-22T06:34:31.012055+0000","last_unstale":"2024-04-22T06:37:39.037085+0000","last_undegraded":"2024-04-22T06:37:39.037085+0000","last_fullsized":"2024-04-22T06:37:39.037085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:31:11.378820+0000","last_clean_scrub_stamp":"2024-04-22T06:31:11.378820+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:03:05.351001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975078+0000","last_change":"2024-04-22T06:36:45.784112+0000","last_active":"2024-04-22T06:37:36.975078+0000","last_peered":"2024-04-22T06:37:36.975078+0000","last_clean":"2024-04-22T06:37:36.975078+0000","last_became_active":"2024-04-22T06:36:45.783569+0000","last_became_peered":"2024-04-22T06:36:45.783569+0000","last_unstale":"2024-04-22T06:37:36.975078+0000","last_undegraded":"2024-04-22T06:37:36.975078+0000","last_fullsized":"2024-04-22T06:37:36.975078+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:45:37.417280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934281+0000","last_change":"2024-04-22T06:35:39.800810+0000","last_active":"2024-04-22T06:37:36.934281+0000","last_peered":"2024-04-22T06:37:36.934281+0000","last_clean":"2024-04-22T06:37:36.934281+0000","last_became_active":"2024-04-22T06:35:39.800459+0000","last_became_peered":"2024-04-22T06:35:39.800459+0000","last_unstale":"2024-04-22T06:37:36.934281+0000","last_undegraded":"2024-04-22T06:37:36.934281+0000","last_fullsized":"2024-04-22T06:37:36.934281+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:34:15.684976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933385+0000","last_change":"2024-04-22T06:35:39.802517+0000","last_active":"2024-04-22T06:37:36.933385+0000","last_peered":"2024-04-22T06:37:36.933385+0000","last_clean":"2024-04-22T06:37:36.933385+0000","last_became_active":"2024-04-22T06:35:39.802200+0000","last_became_peered":"2024-04-22T06:35:39.802200+0000","last_unstale":"2024-04-22T06:37:36.933385+0000","last_undegraded":"2024-04-22T06:37:36.933385+0000","last_fullsized":"2024-04-22T06:37:36.933385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:33:40.955269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975065+0000","last_change":"2024-04-22T06:35:40.818611+0000","last_active":"2024-04-22T06:37:36.975065+0000","last_peered":"2024-04-22T06:37:36.975065+0000","last_clean":"2024-04-22T06:37:36.975065+0000","last_became_active":"2024-04-22T06:35:40.817436+0000","last_became_peered":"2024-04-22T06:35:40.817436+0000","last_unstale":"2024-04-22T06:37:36.975065+0000","last_undegraded":"2024-04-22T06:37:36.975065+0000","last_fullsized":"2024-04-22T06:37:36.975065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:11:07.483749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'49","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.134795+0000","last_change":"2024-04-22T06:35:39.804316+0000","last_active":"2024-04-22T06:37:37.134795+0000","last_peered":"2024-04-22T06:37:37.134795+0000","last_clean":"2024-04-22T06:37:37.134795+0000","last_became_active":"2024-04-22T06:35:39.803951+0000","last_became_peered":"2024-04-22T06:35:39.803951+0000","last_unstale":"2024-04-22T06:37:37.134795+0000","last_undegraded":"2024-04-22T06:37:37.134795+0000","last_fullsized":"2024-04-22T06:37:37.134795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:14:19.240420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974921+0000","last_change":"2024-04-22T06:35:40.811483+0000","last_active":"2024-04-22T06:37:36.974921+0000","last_peered":"2024-04-22T06:37:36.974921+0000","last_clean":"2024-04-22T06:37:36.974921+0000","last_became_active":"2024-04-22T06:35:40.811175+0000","last_became_peered":"2024-04-22T06:35:40.811175+0000","last_unstale":"2024-04-22T06:37:36.974921+0000","last_undegraded":"2024-04-22T06:37:36.974921+0000","last_fullsized":"2024-04-22T06:37:36.974921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:28:43.879416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975403+0000","last_change":"2024-04-22T06:35:39.810112+0000","last_active":"2024-04-22T06:37:36.975403+0000","last_peered":"2024-04-22T06:37:36.975403+0000","last_clean":"2024-04-22T06:37:36.975403+0000","last_became_active":"2024-04-22T06:35:39.809758+0000","last_became_peered":"2024-04-22T06:35:39.809758+0000","last_unstale":"2024-04-22T06:37:36.975403+0000","last_undegraded":"2024-04-22T06:37:36.975403+0000","last_fullsized":"2024-04-22T06:37:36.975403+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:13:39.054864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975490+0000","last_change":"2024-04-22T06:35:40.811140+0000","last_active":"2024-04-22T06:37:36.975490+0000","last_peered":"2024-04-22T06:37:36.975490+0000","last_clean":"2024-04-22T06:37:36.975490+0000","last_became_active":"2024-04-22T06:35:40.810778+0000","last_became_peered":"2024-04-22T06:35:40.810778+0000","last_unstale":"2024-04-22T06:37:36.975490+0000","last_undegraded":"2024-04-22T06:37:36.975490+0000","last_fullsized":"2024-04-22T06:37:36.975490+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:29:41.530996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975324+0000","last_change":"2024-04-22T06:35:39.801822+0000","last_active":"2024-04-22T06:37:36.975324+0000","last_peered":"2024-04-22T06:37:36.975324+0000","last_clean":"2024-04-22T06:37:36.975324+0000","last_became_active":"2024-04-22T06:35:39.801481+0000","last_became_peered":"2024-04-22T06:35:39.801481+0000","last_unstale":"2024-04-22T06:37:36.975324+0000","last_undegraded":"2024-04-22T06:37:36.975324+0000","last_fullsized":"2024-04-22T06:37:36.975324+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:12:45.499847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.932851+0000","last_change":"2024-04-22T06:35:40.811240+0000","last_active":"2024-04-22T06:37:36.932851+0000","last_peered":"2024-04-22T06:37:36.932851+0000","last_clean":"2024-04-22T06:37:36.932851+0000","last_became_active":"2024-04-22T06:35:40.810916+0000","last_became_peered":"2024-04-22T06:35:40.810916+0000","last_unstale":"2024-04-22T06:37:36.932851+0000","last_undegraded":"2024-04-22T06:37:36.932851+0000","last_fullsized":"2024-04-22T06:37:36.932851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:54:37.020997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974234+0000","last_change":"2024-04-22T06:35:39.809829+0000","last_active":"2024-04-22T06:37:36.974234+0000","last_peered":"2024-04-22T06:37:36.974234+0000","last_clean":"2024-04-22T06:37:36.974234+0000","last_became_active":"2024-04-22T06:35:39.809387+0000","last_became_peered":"2024-04-22T06:35:39.809387+0000","last_unstale":"2024-04-22T06:37:36.974234+0000","last_undegraded":"2024-04-22T06:37:36.974234+0000","last_fullsized":"2024-04-22T06:37:36.974234+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:27:13.843047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999879+0000","last_change":"2024-04-22T06:35:40.806191+0000","last_active":"2024-04-22T06:37:36.999879+0000","last_peered":"2024-04-22T06:37:36.999879+0000","last_clean":"2024-04-22T06:37:36.999879+0000","last_became_active":"2024-04-22T06:35:40.805812+0000","last_became_peered":"2024-04-22T06:35:40.805812+0000","last_unstale":"2024-04-22T06:37:36.999879+0000","last_undegraded":"2024-04-22T06:37:36.999879+0000","last_fullsized":"2024-04-22T06:37:36.999879+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T13:17:18.321805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'4","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974076+0000","last_change":"2024-04-22T06:35:39.809062+0000","last_active":"2024-04-22T06:37:36.974076+0000","last_peered":"2024-04-22T06:37:36.974076+0000","last_clean":"2024-04-22T06:37:36.974076+0000","last_became_active":"2024-04-22T06:35:39.808730+0000","last_became_peered":"2024-04-22T06:35:39.808730+0000","last_unstale":"2024-04-22T06:37:36.974076+0000","last_undegraded":"2024-04-22T06:37:36.974076+0000","last_fullsized":"2024-04-22T06:37:36.974076+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:41:13.165165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974320+0000","last_change":"2024-04-22T06:35:40.810714+0000","last_active":"2024-04-22T06:37:36.974320+0000","last_peered":"2024-04-22T06:37:36.974320+0000","last_clean":"2024-04-22T06:37:36.974320+0000","last_became_active":"2024-04-22T06:35:40.810410+0000","last_became_peered":"2024-04-22T06:35:40.810410+0000","last_unstale":"2024-04-22T06:37:36.974320+0000","last_undegraded":"2024-04-22T06:37:36.974320+0000","last_fullsized":"2024-04-22T06:37:36.974320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:45:24.614141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.933129+0000","last_change":"2024-04-22T06:35:39.803728+0000","last_active":"2024-04-22T06:37:36.933129+0000","last_peered":"2024-04-22T06:37:36.933129+0000","last_clean":"2024-04-22T06:37:36.933129+0000","last_became_active":"2024-04-22T06:35:39.803387+0000","last_became_peered":"2024-04-22T06:35:39.803387+0000","last_unstale":"2024-04-22T06:37:36.933129+0000","last_undegraded":"2024-04-22T06:37:36.933129+0000","last_fullsized":"2024-04-22T06:37:36.933129+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:51:53.362714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":24,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934356+0000","last_change":"2024-04-22T06:35:40.819916+0000","last_active":"2024-04-22T06:37:36.934356+0000","last_peered":"2024-04-22T06:37:36.934356+0000","last_clean":"2024-04-22T06:37:36.934356+0000","last_became_active":"2024-04-22T06:35:40.819445+0000","last_became_peered":"2024-04-22T06:35:40.819445+0000","last_unstale":"2024-04-22T06:37:36.934356+0000","last_undegraded":"2024-04-22T06:37:36.934356+0000","last_fullsized":"2024-04-22T06:37:36.934356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:38:49.599791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'6","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:41.151888+0000","last_change":"2024-04-22T06:35:39.816137+0000","last_active":"2024-04-22T06:37:41.151888+0000","last_peered":"2024-04-22T06:37:41.151888+0000","last_clean":"2024-04-22T06:37:41.151888+0000","last_became_active":"2024-04-22T06:35:39.815771+0000","last_became_peered":"2024-04-22T06:35:39.815771+0000","last_unstale":"2024-04-22T06:37:41.151888+0000","last_undegraded":"2024-04-22T06:37:41.151888+0000","last_fullsized":"2024-04-22T06:37:41.151888+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T18:15:42.690442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934556+0000","last_change":"2024-04-22T06:35:40.814625+0000","last_active":"2024-04-22T06:37:36.934556+0000","last_peered":"2024-04-22T06:37:36.934556+0000","last_clean":"2024-04-22T06:37:36.934556+0000","last_became_active":"2024-04-22T06:35:40.813576+0000","last_became_peered":"2024-04-22T06:35:40.813576+0000","last_unstale":"2024-04-22T06:37:36.934556+0000","last_undegraded":"2024-04-22T06:37:36.934556+0000","last_fullsized":"2024-04-22T06:37:36.934556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:48:59.839657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'41","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.129656+0000","last_change":"2024-04-22T06:35:39.806696+0000","last_active":"2024-04-22T06:37:37.129656+0000","last_peered":"2024-04-22T06:37:37.129656+0000","last_clean":"2024-04-22T06:37:37.129656+0000","last_became_active":"2024-04-22T06:35:39.805964+0000","last_became_peered":"2024-04-22T06:35:39.805964+0000","last_unstale":"2024-04-22T06:37:37.129656+0000","last_undegraded":"2024-04-22T06:37:37.129656+0000","last_fullsized":"2024-04-22T06:37:37.129656+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:32:22.017989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934284+0000","last_change":"2024-04-22T06:35:40.817424+0000","last_active":"2024-04-22T06:37:36.934284+0000","last_peered":"2024-04-22T06:37:36.934284+0000","last_clean":"2024-04-22T06:37:36.934284+0000","last_became_active":"2024-04-22T06:35:40.817038+0000","last_became_peered":"2024-04-22T06:35:40.817038+0000","last_unstale":"2024-04-22T06:37:36.934284+0000","last_undegraded":"2024-04-22T06:37:36.934284+0000","last_fullsized":"2024-04-22T06:37:36.934284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:36:23.378441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975202+0000","last_change":"2024-04-22T06:35:39.808847+0000","last_active":"2024-04-22T06:37:36.975202+0000","last_peered":"2024-04-22T06:37:36.975202+0000","last_clean":"2024-04-22T06:37:36.975202+0000","last_became_active":"2024-04-22T06:35:39.808522+0000","last_became_peered":"2024-04-22T06:35:39.808522+0000","last_unstale":"2024-04-22T06:37:36.975202+0000","last_undegraded":"2024-04-22T06:37:36.975202+0000","last_fullsized":"2024-04-22T06:37:36.975202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:38.778242+0000","last_clean_scrub_stamp":"2024-04-22T06:35:38.778242+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:01:37.392443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.934749+0000","last_change":"2024-04-22T06:35:40.812775+0000","last_active":"2024-04-22T06:37:36.934749+0000","last_peered":"2024-04-22T06:37:36.934749+0000","last_clean":"2024-04-22T06:37:36.934749+0000","last_became_active":"2024-04-22T06:35:40.810879+0000","last_became_peered":"2024-04-22T06:35:40.810879+0000","last_unstale":"2024-04-22T06:37:36.934749+0000","last_undegraded":"2024-04-22T06:37:36.934749+0000","last_fullsized":"2024-04-22T06:37:36.934749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:39:32.559426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999986+0000","last_change":"2024-04-22T06:35:40.818871+0000","last_active":"2024-04-22T06:37:36.999986+0000","last_peered":"2024-04-22T06:37:36.999986+0000","last_clean":"2024-04-22T06:37:36.999986+0000","last_became_active":"2024-04-22T06:35:40.818008+0000","last_became_peered":"2024-04-22T06:35:40.818008+0000","last_unstale":"2024-04-22T06:37:36.999986+0000","last_undegraded":"2024-04-22T06:37:36.999986+0000","last_fullsized":"2024-04-22T06:37:36.999986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:49:04.233243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973870+0000","last_change":"2024-04-22T06:35:40.821771+0000","last_active":"2024-04-22T06:37:36.973870+0000","last_peered":"2024-04-22T06:37:36.973870+0000","last_clean":"2024-04-22T06:37:36.973870+0000","last_became_active":"2024-04-22T06:35:40.820844+0000","last_became_peered":"2024-04-22T06:35:40.820844+0000","last_unstale":"2024-04-22T06:37:36.973870+0000","last_undegraded":"2024-04-22T06:37:36.973870+0000","last_fullsized":"2024-04-22T06:37:36.973870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:32:43.991376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973831+0000","last_change":"2024-04-22T06:35:40.818959+0000","last_active":"2024-04-22T06:37:36.973831+0000","last_peered":"2024-04-22T06:37:36.973831+0000","last_clean":"2024-04-22T06:37:36.973831+0000","last_became_active":"2024-04-22T06:35:40.818645+0000","last_became_peered":"2024-04-22T06:35:40.818645+0000","last_unstale":"2024-04-22T06:37:36.973831+0000","last_undegraded":"2024-04-22T06:37:36.973831+0000","last_fullsized":"2024-04-22T06:37:36.973831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:29:15.119217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975016+0000","last_change":"2024-04-22T06:35:40.806664+0000","last_active":"2024-04-22T06:37:36.975016+0000","last_peered":"2024-04-22T06:37:36.975016+0000","last_clean":"2024-04-22T06:37:36.975016+0000","last_became_active":"2024-04-22T06:35:40.806009+0000","last_became_peered":"2024-04-22T06:35:40.806009+0000","last_unstale":"2024-04-22T06:37:36.975016+0000","last_undegraded":"2024-04-22T06:37:36.975016+0000","last_fullsized":"2024-04-22T06:37:36.975016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T14:58:13.885542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975217+0000","last_change":"2024-04-22T06:35:40.818314+0000","last_active":"2024-04-22T06:37:36.975217+0000","last_peered":"2024-04-22T06:37:36.975217+0000","last_clean":"2024-04-22T06:37:36.975217+0000","last_became_active":"2024-04-22T06:35:40.817811+0000","last_became_peered":"2024-04-22T06:35:40.817811+0000","last_unstale":"2024-04-22T06:37:36.975217+0000","last_undegraded":"2024-04-22T06:37:36.975217+0000","last_fullsized":"2024-04-22T06:37:36.975217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:39:49.639370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:37.000086+0000","last_change":"2024-04-22T06:35:40.810709+0000","last_active":"2024-04-22T06:37:37.000086+0000","last_peered":"2024-04-22T06:37:37.000086+0000","last_clean":"2024-04-22T06:37:37.000086+0000","last_became_active":"2024-04-22T06:35:40.810367+0000","last_became_peered":"2024-04-22T06:35:40.810367+0000","last_unstale":"2024-04-22T06:37:37.000086+0000","last_undegraded":"2024-04-22T06:37:37.000086+0000","last_fullsized":"2024-04-22T06:37:37.000086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:58:37.628659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974443+0000","last_change":"2024-04-22T06:35:40.813559+0000","last_active":"2024-04-22T06:37:36.974443+0000","last_peered":"2024-04-22T06:37:36.974443+0000","last_clean":"2024-04-22T06:37:36.974443+0000","last_became_active":"2024-04-22T06:35:40.812629+0000","last_became_peered":"2024-04-22T06:35:40.812629+0000","last_unstale":"2024-04-22T06:37:36.974443+0000","last_undegraded":"2024-04-22T06:37:36.974443+0000","last_fullsized":"2024-04-22T06:37:36.974443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:42:40.546252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":21,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974807+0000","last_change":"2024-04-22T06:35:40.815459+0000","last_active":"2024-04-22T06:37:36.974807+0000","last_peered":"2024-04-22T06:37:36.974807+0000","last_clean":"2024-04-22T06:37:36.974807+0000","last_became_active":"2024-04-22T06:35:40.815104+0000","last_became_peered":"2024-04-22T06:35:40.815104+0000","last_unstale":"2024-04-22T06:37:36.974807+0000","last_undegraded":"2024-04-22T06:37:36.974807+0000","last_fullsized":"2024-04-22T06:37:36.974807+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T17:50:35.296966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974055+0000","last_change":"2024-04-22T06:35:40.819898+0000","last_active":"2024-04-22T06:37:36.974055+0000","last_peered":"2024-04-22T06:37:36.974055+0000","last_clean":"2024-04-22T06:37:36.974055+0000","last_became_active":"2024-04-22T06:35:40.819334+0000","last_became_peered":"2024-04-22T06:35:40.819334+0000","last_unstale":"2024-04-22T06:37:36.974055+0000","last_undegraded":"2024-04-22T06:37:36.974055+0000","last_fullsized":"2024-04-22T06:37:36.974055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T15:33:21.001372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.973997+0000","last_change":"2024-04-22T06:35:40.815691+0000","last_active":"2024-04-22T06:37:36.973997+0000","last_peered":"2024-04-22T06:37:36.973997+0000","last_clean":"2024-04-22T06:37:36.973997+0000","last_became_active":"2024-04-22T06:35:40.815314+0000","last_became_peered":"2024-04-22T06:35:40.815314+0000","last_unstale":"2024-04-22T06:37:36.973997+0000","last_undegraded":"2024-04-22T06:37:36.973997+0000","last_fullsized":"2024-04-22T06:37:36.973997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:14:46.378979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.975538+0000","last_change":"2024-04-22T06:35:40.818942+0000","last_active":"2024-04-22T06:37:36.975538+0000","last_peered":"2024-04-22T06:37:36.975538+0000","last_clean":"2024-04-22T06:37:36.975538+0000","last_became_active":"2024-04-22T06:35:40.818586+0000","last_became_peered":"2024-04-22T06:35:40.818586+0000","last_unstale":"2024-04-22T06:37:36.975538+0000","last_undegraded":"2024-04-22T06:37:36.975538+0000","last_fullsized":"2024-04-22T06:37:36.975538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:19.563830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.974660+0000","last_change":"2024-04-22T06:35:40.824028+0000","last_active":"2024-04-22T06:37:36.974660+0000","last_peered":"2024-04-22T06:37:36.974660+0000","last_clean":"2024-04-22T06:37:36.974660+0000","last_became_active":"2024-04-22T06:35:40.823523+0000","last_became_peered":"2024-04-22T06:35:40.823523+0000","last_unstale":"2024-04-22T06:37:36.974660+0000","last_undegraded":"2024-04-22T06:37:36.974660+0000","last_fullsized":"2024-04-22T06:37:36.974660+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T12:37:08.259157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-22T06:37:36.999973+0000","last_change":"2024-04-22T06:35:40.807635+0000","last_active":"2024-04-22T06:37:36.999973+0000","last_peered":"2024-04-22T06:37:36.999973+0000","last_clean":"2024-04-22T06:37:36.999973+0000","last_became_active":"2024-04-22T06:35:40.807274+0000","last_became_peered":"2024-04-22T06:35:40.807274+0000","last_unstale":"2024-04-22T06:37:36.999973+0000","last_undegraded":"2024-04-22T06:37:36.999973+0000","last_fullsized":"2024-04-22T06:37:36.999973+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:35:39.786865+0000","last_clean_scrub_stamp":"2024-04-22T06:35:39.786865+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T16:01:34.882710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":602453,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1308,"num_read_kb":772,"num_write":378,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1807359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":305,"ondisk_log_size":305,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":485943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514538,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37148,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708004,"statfs":{"total":95995035648,"available":95956996096,"internally_reserved":0,"allocated":1855488,"data_stored":1277373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1429504,"data_stored":878630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972404,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":879008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1081105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430270,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":485913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":554464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":486861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":555007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1564672,"data_stored":1009772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":486614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37072,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708080,"statfs":{"total":95995035648,"available":95957073920,"internally_reserved":0,"allocated":1777664,"data_stored":1209979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":135202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:37:48.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:49.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- 172.21.15.105:0/814168846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 msgr2=0x7fc92c10dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- 172.21.15.105:0/814168846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c10dc90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc924009cd0 tx=0x7fc92402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- 172.21.15.105:0/814168846 shutdown_connections 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- 172.21.15.105:0/814168846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c10dc90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- 172.21.15.105:0/814168846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c1011a0 0x7fc92c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- 172.21.15.105:0/814168846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c105e30 0x7fc92c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- 172.21.15.105:0/814168846 >> 172.21.15.105:0/814168846 conn(0x7fc92c077e30 msgr2=0x7fc92c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- 172.21.15.105:0/814168846 shutdown_connections 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- 172.21.15.105:0/814168846 wait complete. 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 Processor -- start 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- start start 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 0x7fc92c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c105e30 0x7fc92c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c112200 con 0x7fc92c101b60 2024-04-22T06:37:49.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c112080 con 0x7fc92c105e30 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc9330a5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c112380 con 0x7fc92c1011a0 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.078+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40924/0 (socket says 172.21.15.105:40924) 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 -- 172.21.15.105:0/2010414975 learned_addr learned my addr 172.21.15.105:0/2010414975 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc931642640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c105e30 0x7fc92c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc930e41640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 0x7fc92c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 -- 172.21.15.105:0/2010414975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 msgr2=0x7fc92c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 0x7fc92c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 -- 172.21.15.105:0/2010414975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c105e30 msgr2=0x7fc92c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c105e30 0x7fc92c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 -- 172.21.15.105:0/2010414975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c19d620 con 0x7fc92c101b60 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc930e41640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 0x7fc92c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:49.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc92bfff640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c890 tx=0x7fc91800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918004220 con 0x7fc92c101b60 2024-04-22T06:37:49.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c19d910 con 0x7fc92c101b60 2024-04-22T06:37:49.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9180043c0 con 0x7fc92c101b60 2024-04-22T06:37:49.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918004a00 con 0x7fc92c101b60 2024-04-22T06:37:49.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.079+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c1a51a0 con 0x7fc92c101b60 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.080+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc92c068de0 con 0x7fc92c101b60 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.081+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fc9180079b0 con 0x7fc92c101b60 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.082+0000 7fc929ffb640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 0x7fc91c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.082+0000 7fc930e41640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 0x7fc91c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.082+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc9180d0840 con 0x7fc92c101b60 2024-04-22T06:37:49.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.082+0000 7fc930e41640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 0x7fc91c0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc92000baa0 tx=0x7fc920005d90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.087+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc918014030 con 0x7fc92c101b60 2024-04-22T06:37:49.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.304+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc92c1093b0 con 0x7fc92c101b60 2024-04-22T06:37:49.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.305+0000 7fc929ffb640 1 -- 172.21.15.105:0/2010414975 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fc91809cd00 con 0x7fc92c101b60 2024-04-22T06:37:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:49.304 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:37:49.304 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:37:49.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.306+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 msgr2=0x7fc91c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.306+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 0x7fc91c0a6300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc92000baa0 tx=0x7fc920005d90 comp rx=0 tx=0).stop 2024-04-22T06:37:49.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.306+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 msgr2=0x7fc92c198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.306+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c890 tx=0x7fc91800cd50 comp rx=0 tx=0).stop 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 shutdown_connections 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc91c0a3e40 0x7fc91c0a6300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc92c105e30 0x7fc92c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc92c101b60 0x7fc92c198bc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 --2- 172.21.15.105:0/2010414975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc92c1011a0 0x7fc92c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 >> 172.21.15.105:0/2010414975 conn(0x7fc92c077e30 msgr2=0x7fc92c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 shutdown_connections 2024-04-22T06:37:49.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.307+0000 7fc9330a5640 1 -- 172.21.15.105:0/2010414975 wait complete. 2024-04-22T06:37:49.316 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:49 smithi105 ceph-mon[27916]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.316 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:49 smithi105 ceph-mon[27916]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.317 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 -- 172.21.15.105:0/2856359739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101ad0 msgr2=0x7f443810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2856359739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101ad0 0x7f443810dcb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4430009ca0 tx=0x7f443002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 -- 172.21.15.105:0/2856359739 shutdown_connections 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2856359739 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101ad0 0x7f443810dcb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2856359739 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101130 0x7f4438101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2856359739 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f44381072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 -- 172.21.15.105:0/2856359739 >> 172.21.15.105:0/2856359739 conn(0x7f4438077e30 msgr2=0x7f4438078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 -- 172.21.15.105:0/2856359739 shutdown_connections 2024-04-22T06:37:49.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 -- 172.21.15.105:0/2856359739 wait complete. 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.419+0000 7f443d5a4640 1 Processor -- start 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 -- start start 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101ad0 0x7f4438197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f443819bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f44381110d0 con 0x7f4438101130 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4438110f50 con 0x7f4438106f10 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f443d5a4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4438111250 con 0x7f4438101ad0 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40936/0 (socket says 172.21.15.105:40936) 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 -- 172.21.15.105:0/2965256959 learned_addr learned my addr 172.21.15.105:0/2965256959 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:49.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f44367fc640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101ad0 0x7f4438197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 -- 172.21.15.105:0/2965256959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101ad0 msgr2=0x7f4438197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f44377fe640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f443819bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101ad0 0x7f4438197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 -- 172.21.15.105:0/2965256959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 msgr2=0x7f443819bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f443819bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 -- 172.21.15.105:0/2965256959 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443819c620 con 0x7f4438101130 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f44377fe640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f443819bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f4436ffd640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f442800c890 tx=0x7f442800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.420+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428004220 con 0x7f4438101130 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.421+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44280043c0 con 0x7f4438101130 2024-04-22T06:37:49.423 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.421+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443819c910 con 0x7f4438101130 2024-04-22T06:37:49.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.421+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428004a00 con 0x7f4438101130 2024-04-22T06:37:49.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.421+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44381a41a0 con 0x7f4438101130 2024-04-22T06:37:49.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.421+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4438069230 con 0x7f4438101130 2024-04-22T06:37:49.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.423+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f44280079b0 con 0x7f4438101130 2024-04-22T06:37:49.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.423+0000 7f441ffff640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 0x7f44200a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.423+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f44280d0a60 con 0x7f4438101130 2024-04-22T06:37:49.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.423+0000 7f44367fc640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 0x7f44200a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.424+0000 7f44367fc640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 0x7f44200a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4438106670 tx=0x7f442c00a2d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.426 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.427+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4428014030 con 0x7f4438101130 2024-04-22T06:37:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:49 smithi112 ceph-mon[31236]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:49 smithi112 ceph-mon[31236]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:49 smithi177 ceph-mon[31346]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:49 smithi177 ceph-mon[31346]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:49.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.609+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4438198980 con 0x7f4438101130 2024-04-22T06:37:49.610 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.611+0000 7f441ffff640 1 -- 172.21.15.105:0/2965256959 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7f442809cf20 con 0x7f4438101130 2024-04-22T06:37:49.611 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:49.611 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 msgr2=0x7f44200a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 0x7f44200a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4438106670 tx=0x7f442c00a2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 msgr2=0x7f4438197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f442800c890 tx=0x7f442800cc90 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 shutdown_connections 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f44200a3e40 0x7f44200a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4438106f10 0x7f443819bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4438101ad0 0x7f4438197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 --2- 172.21.15.105:0/2965256959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4438101130 0x7f4438197630 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 >> 172.21.15.105:0/2965256959 conn(0x7f4438077e30 msgr2=0x7f44380ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 shutdown_connections 2024-04-22T06:37:49.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.613+0000 7f443d5a4640 1 -- 172.21.15.105:0/2965256959 wait complete. 2024-04-22T06:37:49.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:37:49.627 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:37:49.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 -- 172.21.15.105:0/1222829905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac1031d0 msgr2=0x7f80ac1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.783 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 --2- 172.21.15.105:0/1222829905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac1031d0 0x7f80ac1035b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f809c009ca0 tx=0x7f809c02c230 comp rx=0 tx=0).stop 2024-04-22T06:37:49.783 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 -- 172.21.15.105:0/1222829905 shutdown_connections 2024-04-22T06:37:49.783 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 --2- 172.21.15.105:0/1222829905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac10c510 0x7f80ac10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.783 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 --2- 172.21.15.105:0/1222829905 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac103af0 0x7f80ac10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 --2- 172.21.15.105:0/1222829905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac1031d0 0x7f80ac1035b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.732+0000 7f80b13f4640 1 -- 172.21.15.105:0/1222829905 >> 172.21.15.105:0/1222829905 conn(0x7f80ac077f50 msgr2=0x7f80ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.733+0000 7f80b13f4640 1 -- 172.21.15.105:0/1222829905 shutdown_connections 2024-04-22T06:37:49.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.733+0000 7f80b13f4640 1 -- 172.21.15.105:0/1222829905 wait complete. 2024-04-22T06:37:49.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.733+0000 7f80b13f4640 1 Processor -- start 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.733+0000 7f80b13f4640 1 -- start start 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac1031d0 0x7f80ac1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac10c510 0x7f80ac19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac112180 con 0x7f80ac103af0 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac112000 con 0x7f80ac1031d0 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f80ac112300 con 0x7f80ac10c510 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40956/0 (socket says 172.21.15.105:40956) 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 -- 172.21.15.105:0/3964332624 learned_addr learned my addr 172.21.15.105:0/3964332624 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80aaffd640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac1031d0 0x7f80ac1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80ab7fe640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac10c510 0x7f80ac19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 -- 172.21.15.105:0/3964332624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac10c510 msgr2=0x7f80ac19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac10c510 0x7f80ac19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 -- 172.21.15.105:0/3964332624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac1031d0 msgr2=0x7f80ac1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac1031d0 0x7f80ac1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 -- 172.21.15.105:0/3964332624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ac19d6d0 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a3fff640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f809400c920 tx=0x7f809400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094007c70 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8094007e10 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ac19d9c0 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094018820 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.734+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ac1a5250 con 0x7f80ac103af0 2024-04-22T06:37:49.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.736+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ac068d70 con 0x7f80ac103af0 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.737+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f809400f450 con 0x7f80ac103af0 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.737+0000 7f80a8ff9640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 0x7f80980a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.738+0000 7f80aaffd640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 0x7f80980a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.738+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f80940d0c20 con 0x7f80ac103af0 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.738+0000 7f80aaffd640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 0x7f80980a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f809c037040 tx=0x7f809c032340 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:49.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.742+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8094014030 con 0x7f80ac103af0 2024-04-22T06:37:49.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.926+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80ac02d5c0 con 0x7f80ac103af0 2024-04-22T06:37:49.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.926+0000 7f80a8ff9640 1 -- 172.21.15.105:0/3964332624 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f809400de20 con 0x7f80ac103af0 2024-04-22T06:37:49.926 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 msgr2=0x7f80980a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 0x7f80980a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f809c037040 tx=0x7f809c032340 comp rx=0 tx=0).stop 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 msgr2=0x7f80ac198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f809400c920 tx=0x7f809400cde0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 shutdown_connections 2024-04-22T06:37:49.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f80980a3ec0 0x7f80980a6380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f80ac10c510 0x7f80ac19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f80ac103af0 0x7f80ac198c20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 --2- 172.21.15.105:0/3964332624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80ac1031d0 0x7f80ac1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 >> 172.21.15.105:0/3964332624 conn(0x7f80ac077f50 msgr2=0x7f80ac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 shutdown_connections 2024-04-22T06:37:49.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:49.928+0000 7f80b13f4640 1 -- 172.21.15.105:0/3964332624 wait complete. 2024-04-22T06:37:49.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:37:50.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- 172.21.15.105:0/3809659165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f81031d0 msgr2=0x7f93f81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 --2- 172.21.15.105:0/3809659165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f81031d0 0x7f93f81035b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f93ec009900 tx=0x7f93ec02bd60 comp rx=0 tx=0).stop 2024-04-22T06:37:50.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- 172.21.15.105:0/3809659165 shutdown_connections 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 --2- 172.21.15.105:0/3809659165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f810c510 0x7f93f810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 --2- 172.21.15.105:0/3809659165 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 0x7f93f810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 --2- 172.21.15.105:0/3809659165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f81031d0 0x7f93f81035b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- 172.21.15.105:0/3809659165 >> 172.21.15.105:0/3809659165 conn(0x7f93f8077f50 msgr2=0x7f93f8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- 172.21.15.105:0/3809659165 shutdown_connections 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- 172.21.15.105:0/3809659165 wait complete. 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 Processor -- start 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.048+0000 7f93ff06a640 1 -- start start 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f81031d0 0x7f93f819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 0x7f93f819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81121f0 con 0x7f93f810c510 2024-04-22T06:37:50.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8112070 con 0x7f93f81031d0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8112370 con 0x7f93f8103af0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40978/0 (socket says 172.21.15.105:40978) 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 -- 172.21.15.105:0/2339964098 learned_addr learned my addr 172.21.15.105:0/2339964098 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93f7fff640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 0x7f93f819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fce06640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f81031d0 0x7f93f819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 -- 172.21.15.105:0/2339964098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 msgr2=0x7f93f819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 0x7f93f819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 -- 172.21.15.105:0/2339964098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f81031d0 msgr2=0x7f93f819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f81031d0 0x7f93f819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 -- 172.21.15.105:0/2339964098 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f8198d10 con 0x7f93f810c510 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93fd607640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f93f000eb10 tx=0x7f93f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0004b20 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f8199000 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93f0004cc0 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.049+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0010670 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.050+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f81a52a0 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.050+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f8068d70 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.051+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f93f0005020 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.052+0000 7f93f5ffb640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 0x7f93e80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.052+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f93f00d1080 con 0x7f93f810c510 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.052+0000 7f93fce06640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 0x7f93e80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.053+0000 7f93fce06640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 0x7f93e80a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93ec006010 tx=0x7f93ec000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.056+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93f009d540 con 0x7f93f810c510 2024-04-22T06:37:50.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.241+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f93f8107690 con 0x7f93f810c510 2024-04-22T06:37:50.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.241+0000 7f93f5ffb640 1 -- 172.21.15.105:0/2339964098 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f93f009ce40 con 0x7f93f810c510 2024-04-22T06:37:50.241 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 msgr2=0x7f93e80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 0x7f93e80a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93ec006010 tx=0x7f93ec000bc0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 msgr2=0x7f93f8198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f93f000eb10 tx=0x7f93f000c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 shutdown_connections 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f93e80a3e40 0x7f93e80a6300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93f810c510 0x7f93f8198590 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f93f8103af0 0x7f93f819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 --2- 172.21.15.105:0/2339964098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93f81031d0 0x7f93f819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.243+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 >> 172.21.15.105:0/2339964098 conn(0x7f93f8077f50 msgr2=0x7f93f80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.244+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 shutdown_connections 2024-04-22T06:37:50.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.244+0000 7f93ff06a640 1 -- 172.21.15.105:0/2339964098 wait complete. 2024-04-22T06:37:50.253 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:50 smithi105 ceph-mon[27916]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:50.253 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:50 smithi105 ceph-mon[27916]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-04-22T06:37:50.253 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2010414975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.253 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2965256959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.253 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:50 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3964332624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:50.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- 172.21.15.105:0/2372928446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d010e7f0 msgr2=0x7fc8d0110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/2372928446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d010e7f0 0x7fc8d0110be0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009ca0 tx=0x7fc8c802c180 comp rx=0 tx=0).stop 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- 172.21.15.105:0/2372928446 shutdown_connections 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/2372928446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d010e7f0 0x7fc8d0110be0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/2372928446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d0074c20 0x7fc8d010e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/2372928446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0074300 0x7fc8d00746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- 172.21.15.105:0/2372928446 >> 172.21.15.105:0/2372928446 conn(0x7fc8d0079350 msgr2=0x7fc8d0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- 172.21.15.105:0/2372928446 shutdown_connections 2024-04-22T06:37:50.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- 172.21.15.105:0/2372928446 wait complete. 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 Processor -- start 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 -- start start 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.357+0000 7fc8d7d04640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0074c20 0x7fc8d01a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 0x7fc8d019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0114380 con 0x7fc8d0074300 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0114200 con 0x7fc8d010e7f0 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0114500 con 0x7fc8d0074c20 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41000/0 (socket says 172.21.15.105:41000) 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 -- 172.21.15.105:0/1399106381 learned_addr learned my addr 172.21.15.105:0/1399106381 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d62a1640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 0x7fc8d019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 -- 172.21.15.105:0/1399106381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0074c20 msgr2=0x7fc8d01a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0074c20 0x7fc8d01a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 -- 172.21.15.105:0/1399106381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 msgr2=0x7fc8d019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 0x7fc8d019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 -- 172.21.15.105:0/1399106381 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d019b020 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d62a1640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 0x7fc8d019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d5aa0640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000d950 tx=0x7fc8c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0017030 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d019b310 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8c0004020 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00047a0 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.358+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01a7620 con 0x7fc8d0074300 2024-04-22T06:37:50.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.359+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d00759b0 con 0x7fc8d0074300 2024-04-22T06:37:50.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.360+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fc8c0009a30 con 0x7fc8d0074300 2024-04-22T06:37:50.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.361+0000 7fc8c6ffd640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 0x7fc8bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.361+0000 7fc8d529f640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 0x7fc8bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.361+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc8c00d0bb0 con 0x7fc8d0074300 2024-04-22T06:37:50.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.362+0000 7fc8d529f640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 0x7fc8bc0a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005d80 tx=0x7fc8cc005c90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.369+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8c009d070 con 0x7fc8d0074300 2024-04-22T06:37:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:50 smithi112 ceph-mon[31236]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:50.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:50 smithi112 ceph-mon[31236]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-04-22T06:37:50.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2010414975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2965256959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:50 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3964332624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:50 smithi177 ceph-mon[31346]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T06:37:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:50 smithi177 ceph-mon[31346]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 56 op/s 2024-04-22T06:37:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2010414975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2965256959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:50.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:50 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3964332624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:37:50.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.554+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8d0068d70 con 0x7fc8d0074300 2024-04-22T06:37:50.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.555+0000 7fc8c6ffd640 1 -- 172.21.15.105:0/1399106381 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fc8c009c970 con 0x7fc8d0074300 2024-04-22T06:37:50.555 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 msgr2=0x7fc8bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 0x7fc8bc0a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005d80 tx=0x7fc8cc005c90 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 msgr2=0x7fc8d01a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000d950 tx=0x7fc8c000de10 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 shutdown_connections 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc8bc0a3e40 0x7fc8bc0a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc8d010e7f0 0x7fc8d019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8d0074c20 0x7fc8d01a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 --2- 172.21.15.105:0/1399106381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8d0074300 0x7fc8d01a06a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.557 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 >> 172.21.15.105:0/1399106381 conn(0x7fc8d0079350 msgr2=0x7fc8d00fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 shutdown_connections 2024-04-22T06:37:50.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.557+0000 7fc8d7d04640 1 -- 172.21.15.105:0/1399106381 wait complete. 2024-04-22T06:37:50.568 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:50.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 -- 172.21.15.105:0/2559675296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b8105ee0 msgr2=0x7f81b81062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 --2- 172.21.15.105:0/2559675296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b8105ee0 0x7f81b81062c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009900 tx=0x7f81a802bc80 comp rx=0 tx=0).stop 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 -- 172.21.15.105:0/2559675296 shutdown_connections 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 --2- 172.21.15.105:0/2559675296 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 0x7f81b810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 --2- 172.21.15.105:0/2559675296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b81011a0 0x7f81b8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 --2- 172.21.15.105:0/2559675296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b8105ee0 0x7f81b81062c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 -- 172.21.15.105:0/2559675296 >> 172.21.15.105:0/2559675296 conn(0x7f81b8077c20 msgr2=0x7f81b8078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 -- 172.21.15.105:0/2559675296 shutdown_connections 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 -- 172.21.15.105:0/2559675296 wait complete. 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.667+0000 7f81bd481640 1 Processor -- start 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 -- start start 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 0x7f81b81a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b8105ee0 0x7f81b819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81b8112270 con 0x7f81b81011a0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f81b81120f0 con 0x7f81b8105ee0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81bd481640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f81b81123f0 con 0x7f81b8101b60 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41022/0 (socket says 172.21.15.105:41022) 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 -- 172.21.15.105:0/3134273418 learned_addr learned my addr 172.21.15.105:0/3134273418 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b77fe640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b8105ee0 0x7f81b819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b67fc640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 0x7f81b81a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 -- 172.21.15.105:0/3134273418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 msgr2=0x7f81b81a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 0x7f81b81a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 -- 172.21.15.105:0/3134273418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b8105ee0 msgr2=0x7f81b819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b8105ee0 0x7f81b819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 -- 172.21.15.105:0/3134273418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b819b040 con 0x7f81b81011a0 2024-04-22T06:37:50.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f81b6ffd640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81a8005940 tx=0x7f81a80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.668+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8043020 con 0x7f81b81011a0 2024-04-22T06:37:50.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.669+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b819b2d0 con 0x7f81b81011a0 2024-04-22T06:37:50.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.669+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81a802cae0 con 0x7f81b81011a0 2024-04-22T06:37:50.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.669+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8003af0 con 0x7f81b81011a0 2024-04-22T06:37:50.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.669+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b81a75f0 con 0x7f81b81011a0 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.670+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b819b640 con 0x7f81b81011a0 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.671+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f81a8046020 con 0x7f81b81011a0 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.671+0000 7f819ffff640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 0x7f81a00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.671+0000 7f81b67fc640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 0x7f81a00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.671+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f81a80f29e0 con 0x7f81b81011a0 2024-04-22T06:37:50.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.672+0000 7f81b67fc640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 0x7f81a00a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f81ac005fd0 tx=0x7f81ac005850 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:50.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.675+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81a804d020 con 0x7f81b81011a0 2024-04-22T06:37:50.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.892+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81b819bc60 con 0x7f81b81011a0 2024-04-22T06:37:50.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.892+0000 7f819ffff640 1 -- 172.21.15.105:0/3134273418 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f81a80befb0 con 0x7f81b81011a0 2024-04-22T06:37:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:50.892 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:37:50.892 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 msgr2=0x7f81a00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 0x7f81a00a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f81ac005fd0 tx=0x7f81ac005850 comp rx=0 tx=0).stop 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 msgr2=0x7f81b81a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81a8005940 tx=0x7f81a80045e0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 shutdown_connections 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f81a00a3e40 0x7f81a00a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f81b8105ee0 0x7f81b819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.894 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f81b8101b60 0x7f81b81a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 --2- 172.21.15.105:0/3134273418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81b81011a0 0x7f81b81a0670 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:50.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.894+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 >> 172.21.15.105:0/3134273418 conn(0x7f81b8077c20 msgr2=0x7f81b810c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:50.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.895+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 shutdown_connections 2024-04-22T06:37:50.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:50.895+0000 7f81bd481640 1 -- 172.21.15.105:0/3134273418 wait complete. 2024-04-22T06:37:50.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:37:51.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 -- 172.21.15.105:0/2427595919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0240107340 msgr2=0x7f024010eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 --2- 172.21.15.105:0/2427595919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0240107340 0x7f024010eec0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0238009ca0 tx=0x7f023802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 -- 172.21.15.105:0/2427595919 shutdown_connections 2024-04-22T06:37:51.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 --2- 172.21.15.105:0/2427595919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0240107340 0x7f024010eec0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 --2- 172.21.15.105:0/2427595919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f024006b260 0x7f0240106c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 --2- 172.21.15.105:0/2427595919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f024006a8b0 0x7f024006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 -- 172.21.15.105:0/2427595919 >> 172.21.15.105:0/2427595919 conn(0x7f0240077e30 msgr2=0x7f0240078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 -- 172.21.15.105:0/2427595919 shutdown_connections 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.003+0000 7f024694b640 1 -- 172.21.15.105:0/2427595919 wait complete. 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 Processor -- start 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 -- start start 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 0x7f024019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02401985a0 0x7f0240198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0240112260 con 0x7f024006a8b0 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02401120e0 con 0x7f02401985a0 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.004+0000 7f024694b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f02401123e0 con 0x7f0240107340 2024-04-22T06:37:51.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41034/0 (socket says 172.21.15.105:41034) 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 -- 172.21.15.105:0/3994201047 learned_addr learned my addr 172.21.15.105:0/3994201047 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023f7fe640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 0x7f024019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f0244ee8640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02401985a0 0x7f0240198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 -- 172.21.15.105:0/3994201047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 msgr2=0x7f024019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 0x7f024019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 -- 172.21.15.105:0/3994201047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02401985a0 msgr2=0x7f0240198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02401985a0 0x7f0240198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 -- 172.21.15.105:0/3994201047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02401990c0 con 0x7f024006a8b0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023f7fe640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 0x7f024019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023ffff640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f023000e9f0 tx=0x7f023000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023000cd80 con 0x7f024006a8b0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02300108e0 con 0x7f024006a8b0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02401993b0 con 0x7f024006a8b0 2024-04-22T06:37:51.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230018980 con 0x7f024006a8b0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.005+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02401a5600 con 0x7f024006a8b0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.006+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02401036f0 con 0x7f024006a8b0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.007+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f0230010430 con 0x7f024006a8b0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.008+0000 7f023d7fa640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 0x7f02280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.008+0000 7f023f7fe640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 0x7f02280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.008+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f02300d1330 con 0x7f024006a8b0 2024-04-22T06:37:51.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.008+0000 7f023f7fe640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 0x7f02280a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0240100dc0 tx=0x7f0234005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.011+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f023009d7f0 con 0x7f024006a8b0 2024-04-22T06:37:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2339964098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1399106381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:51 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3134273418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:51.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.193+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f024010a460 con 0x7f024006a8b0 2024-04-22T06:37:51.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.195+0000 7f023d7fa640 1 -- 172.21.15.105:0/3994201047 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22989 (secure 0 0 0) 0x7f023009d0f0 con 0x7f024006a8b0 2024-04-22T06:37:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:51.195 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":92,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:37:36.925038+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:37:51.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 msgr2=0x7f02280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 0x7f02280a6300 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0240100dc0 tx=0x7f0234005f90 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 msgr2=0x7f024019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f023000e9f0 tx=0x7f023000eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 shutdown_connections 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f02280a3e40 0x7f02280a6300 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02401985a0 0x7f0240198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0240107340 0x7f024019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 --2- 172.21.15.105:0/3994201047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f024006a8b0 0x7f024019e3f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 >> 172.21.15.105:0/3994201047 conn(0x7f0240077e30 msgr2=0x7f02400ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 shutdown_connections 2024-04-22T06:37:51.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.197+0000 7f024694b640 1 -- 172.21.15.105:0/3994201047 wait complete. 2024-04-22T06:37:51.208 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 946583 2024-04-22T06:37:51.208 DEBUG:tasks.quiescer:Yielding 2024-04-22T06:37:51.208 INFO:teuthology.run_tasks:Running task workunit... 2024-04-22T06:37:51.224 INFO:tasks.workunit:Pulling workunits from ref 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:37:51.225 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-22T06:37:51.225 INFO:tasks.workunit:timeout=3h 2024-04-22T06:37:51.225 INFO:tasks.workunit:cleanup=True 2024-04-22T06:37:51.225 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:51.226 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:37:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-22 06:35:46.415836565 +0000 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-22 06:35:46.415836565 +0000 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-22 06:37:34.894230137 +0000 2024-04-22T06:37:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-04-22 06:35:46.415836565 +0000 2024-04-22T06:37:51.241 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-22T06:37:51.242 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-22T06:37:51.314 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 86f7587a5a09af35f0895e1a2d08527638fae697 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 -- 172.21.15.105:0/724439381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404102b50 msgr2=0x7f3404106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 --2- 172.21.15.105:0/724439381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404102b50 0x7f3404106b90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009900 tx=0x7f33f402bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 -- 172.21.15.105:0/724439381 shutdown_connections 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 --2- 172.21.15.105:0/724439381 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3404107270 0x7f340410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 --2- 172.21.15.105:0/724439381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404102b50 0x7f3404106b90 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 --2- 172.21.15.105:0/724439381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 0x7f3404102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 -- 172.21.15.105:0/724439381 >> 172.21.15.105:0/724439381 conn(0x7f34040793a0 msgr2=0x7f34040797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 -- 172.21.15.105:0/724439381 shutdown_connections 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 -- 172.21.15.105:0/724439381 wait complete. 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.327+0000 7f34090a0640 1 Processor -- start 2024-04-22T06:37:51.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 -- start start 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 0x7f3404198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 0x7f340419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3404112190 con 0x7f3404107270 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3404112010 con 0x7f34041021a0 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f34090a0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3404112310 con 0x7f340419cfd0 2024-04-22T06:37:51.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41050/0 (socket says 172.21.15.105:41050) 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 -- 172.21.15.105:0/2390683649 learned_addr learned my addr 172.21.15.105:0/2390683649 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f3402d9d640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 0x7f3404198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340359e640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 0x7f340419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 -- 172.21.15.105:0/2390683649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 msgr2=0x7f340419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 0x7f340419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 -- 172.21.15.105:0/2390683649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 msgr2=0x7f3404198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 0x7f3404198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 -- 172.21.15.105:0/2390683649 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340419dac0 con 0x7f3404107270 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340359e640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 0x7f340419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.328+0000 7f340259c640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f33f4036040 tx=0x7f33f402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.329+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4043020 con 0x7f3404107270 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.329+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340419dd50 con 0x7f3404107270 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.329+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33f402cbe0 con 0x7f3404107270 2024-04-22T06:37:51.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.329+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4003bf0 con 0x7f3404107270 2024-04-22T06:37:51.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.329+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34041a55a0 con 0x7f3404107270 2024-04-22T06:37:51.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.330+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3404069230 con 0x7f3404107270 2024-04-22T06:37:51.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.331+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f33f4046020 con 0x7f3404107270 2024-04-22T06:37:51.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.332+0000 7f33ebfff640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 0x7f33ec0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.332+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f33f40f31f0 con 0x7f3404107270 2024-04-22T06:37:51.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.332+0000 7f3402d9d640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 0x7f33ec0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.332+0000 7f3402d9d640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 0x7f33ec0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33f800bbe0 tx=0x7f33f800b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.336+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33f404d020 con 0x7f3404107270 2024-04-22T06:37:51.341 INFO:tasks.workunit.client.0.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-22T06:37:51.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:51 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2339964098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:51 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1399106381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:51 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3134273418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:51 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2339964098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:51 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1399106381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:37:51.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:51 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3134273418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:51.561 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.562+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3404199a60 con 0x7f3404107270 2024-04-22T06:37:51.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.563+0000 7f33ebfff640 1 -- 172.21.15.105:0/2390683649 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f3404199a60 con 0x7f3404107270 2024-04-22T06:37:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:37:51.562 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:37:51.563 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:37:51.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 msgr2=0x7f33ec0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 0x7f33ec0a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33f800bbe0 tx=0x7f33f800b390 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 msgr2=0x7f3404198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f33f4036040 tx=0x7f33f402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 shutdown_connections 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f33ec0a3e40 0x7f33ec0a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f340419cfd0 0x7f340419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3404107270 0x7f3404198c50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 --2- 172.21.15.105:0/2390683649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34041021a0 0x7f3404198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 >> 172.21.15.105:0/2390683649 conn(0x7f34040793a0 msgr2=0x7f34040fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 shutdown_connections 2024-04-22T06:37:51.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.565+0000 7f34090a0640 1 -- 172.21.15.105:0/2390683649 wait complete. 2024-04-22T06:37:51.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-22T06:35:40.134334+0000', 'modified': '2024-04-22T06:37:35.576345+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24472, 'mds_2': 24461, 'mds_3': 24454, 'mds_4': 14640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14622': {'gid': 14622, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6839/256609249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 256609249}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 256609249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14640': {'gid': 14640, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.105:6841/2147123680', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6840', 'nonce': 2147123680}, {'type': 'v1', 'addr': '172.21.15.105:6841', 'nonce': 2147123680}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24397': {'gid': 24397, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6833/624977252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6832', 'nonce': 624977252}, {'type': 'v1', 'addr': '172.21.15.177:6833', 'nonce': 624977252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24398': {'gid': 24398, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/583171810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 583171810}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 583171810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/3883368202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 3883368202}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 3883368202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.177:6835/3449982583', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6834', 'nonce': 3449982583}, {'type': 'v1', 'addr': '172.21.15.177:6835', 'nonce': 3449982583}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.177:6837/4114728122', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6836', 'nonce': 4114728122}, {'type': 'v1', 'addr': '172.21.15.177:6837', 'nonce': 4114728122}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6837/3970644798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3970644798}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3970644798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.177:6839/1381266768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6838', 'nonce': 1381266768}, {'type': 'v1', 'addr': '172.21.15.177:6839', 'nonce': 1381266768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.112:6839/4112363218', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 4112363218}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 4112363218}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [14640, 24454, 24461, 24472, 24500]} 2024-04-22T06:37:51.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-22T06:37:51.576 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-22T06:37:51.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 -- 172.21.15.105:0/1933257204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc102bc0 msgr2=0x7f35cc106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/1933257204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc102bc0 0x7f35cc106c00 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009900 tx=0x7f35b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 -- 172.21.15.105:0/1933257204 shutdown_connections 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/1933257204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 0x7f35cc10eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/1933257204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc102bc0 0x7f35cc106c00 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/1933257204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 0x7f35cc1025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 -- 172.21.15.105:0/1933257204 >> 172.21.15.105:0/1933257204 conn(0x7f35cc0fdab0 msgr2=0x7f35cc0ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 -- 172.21.15.105:0/1933257204 shutdown_connections 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.718+0000 7f35d2aa0640 1 -- 172.21.15.105:0/1933257204 wait complete. 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 Processor -- start 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 -- start start 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 0x7f35cc19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 0x7f35cc19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc112220 con 0x7f35cc198730 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1120a0 con 0x7f35cc102210 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d2aa0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1123a0 con 0x7f35cc1072e0 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41064/0 (socket says 172.21.15.105:41064) 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 -- 172.21.15.105:0/2582629410 learned_addr learned my addr 172.21.15.105:0/2582629410 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d083c640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 0x7f35cc19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35cbfff640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 0x7f35cc19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 -- 172.21.15.105:0/2582629410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 msgr2=0x7f35cc19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 0x7f35cc19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 -- 172.21.15.105:0/2582629410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 msgr2=0x7f35cc19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 0x7f35cc19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.719+0000 7f35d103d640 1 -- 172.21.15.105:0/2582629410 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc199310 con 0x7f35cc198730 2024-04-22T06:37:51.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35d103d640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f35c400eb10 tx=0x7f35c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35c9ffb640 1 -- 172.21.15.105:0/2582629410 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4004b20 con 0x7f35cc198730 2024-04-22T06:37:51.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc1a53f0 con 0x7f35cc198730 2024-04-22T06:37:51.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35c9ffb640 1 -- 172.21.15.105:0/2582629410 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c4004cc0 con 0x7f35cc198730 2024-04-22T06:37:51.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35c9ffb640 1 -- 172.21.15.105:0/2582629410 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4010690 con 0x7f35cc198730 2024-04-22T06:37:51.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.720+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1a57a0 con 0x7f35cc198730 2024-04-22T06:37:51.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.722+0000 7f35c9ffb640 1 -- 172.21.15.105:0/2582629410 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f35c4005020 con 0x7f35cc198730 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.722+0000 7f35c9ffb640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 0x7f35bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d083c640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 0x7f35bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35c9ffb640 1 -- 172.21.15.105:0/2582629410 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f35c40d0eb0 con 0x7f35cc198730 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d083c640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 0x7f35bc0a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005d00 tx=0x7f35c000b410 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 Processor -- start 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 -- start start 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c803710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 0x55b67c806130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b67c75eb10 con 0x55b67c7fa7b0 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b67c6fc240 con 0x55b67c803ce0 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.723+0000 7f35d5768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b67c75aad0 con 0x55b67c7fb160 2024-04-22T06:37:51.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41070/0 (socket says 172.21.15.105:41070) 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 -- 172.21.15.105:0/1376556152 learned_addr learned my addr 172.21.15.105:0/1376556152 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d103d640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 0x55b67c806130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35cbfff640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c803710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 -- 172.21.15.105:0/1376556152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 msgr2=0x55b67c803710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c803710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 -- 172.21.15.105:0/1376556152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 msgr2=0x55b67c806130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 0x55b67c806130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 -- 172.21.15.105:0/1376556152 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b67c4712d0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d103d640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 0x55b67c806130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35cbfff640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c803710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35d083c640 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f35c0011010 tx=0x7f35c0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b6159387ad111c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35b6ffd640 1 -- 172.21.15.105:0/1376556152 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c001e620 con 0x55b67c7fa7b0 2024-04-22T06:37:51.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35b6ffd640 1 -- 172.21.15.105:0/1376556152 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c001ec70 con 0x55b67c7fa7b0 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.724+0000 7f35b6ffd640 1 -- 172.21.15.105:0/1376556152 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0026be0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- 172.21.15.105:0/1376556152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 msgr2=0x55b67c7fab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f35c0011010 tx=0x7f35c0013000 comp rx=0 tx=0).stop 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- 172.21.15.105:0/1376556152 shutdown_connections 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c803ce0 0x55b67c806130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c803710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 --2- 172.21.15.105:0/1376556152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c7fab90 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- 172.21.15.105:0/1376556152 >> 172.21.15.105:0/1376556152 conn(0x55b67c7f4a60 msgr2=0x55b67c7f4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- 172.21.15.105:0/1376556152 shutdown_connections 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- 172.21.15.105:0/1376556152 wait complete. 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 -- start start 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.725+0000 7f35d5768740 1 Processor -- start 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c800cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c801230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b67c806c50 con 0x55b67c803ce0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b67c6e7190 con 0x55b67c7fa7b0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b67c806dd0 con 0x55b67c7fb160 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d5768740 10 client.0 _ll_register_callbacks cb 0x55b67c6e28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41084/0 (socket says 172.21.15.105:41084) 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 -- 172.21.15.105:0/3188367192 learned_addr learned my addr 172.21.15.105:0/3188367192 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35cbfff640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c801230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d083c640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c800cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 -- 172.21.15.105:0/3188367192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 msgr2=0x55b67c801230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c801230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 -- 172.21.15.105:0/3188367192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 msgr2=0x55b67c800cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c800cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 -- 172.21.15.105:0/3188367192 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b67c53b6b0 con 0x55b67c803ce0 2024-04-22T06:37:51.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d083c640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c800cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35d103d640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f35c4010880 tx=0x7f35c409c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292a0be8ea4059e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35abfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.726+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c409d0d0 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f35c40a5450 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c001ec70 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35c40bd2d0 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 10 client.15732 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b67c812ec0 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c001ec70 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f35c40ce7d0 con 0x55b67c803ce0 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 10 client.15732 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 10 client.15732 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 20 client.15732 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:51.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c815420 0x55b67c816890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 4 client.15732 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b67c76c530 con 0x55b67c815420 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.727+0000 7f35d083c640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c815420 0x55b67c816890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35abfff640 10 client.15732 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d083c640 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c815420 0x55b67c816890 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae8e50f54893c7b5 server_cookie=6e3b5111b7a31cdf in_seq=0 out_seq=0 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35abfff640 1 -- 172.21.15.105:0/3188367192 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b67c76c530 con 0x55b67c815420 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35abfff640 10 client.15732 handle_command_reply: tid=0 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 client.15732 shutdown 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 msgr2=0x55b67c801770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f35c4010880 tx=0x7f35c409c870 comp rx=0 tx=0).stop 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 shutdown_connections 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c803ce0 0x55b67c801770 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c815420 0x55b67c816890 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c801230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.734 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.728+0000 7f35d5768740 1 --2- 172.21.15.105:0/3188367192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c800cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.729+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 >> 172.21.15.105:0/3188367192 conn(0x55b67c7f4a60 msgr2=0x55b67c7f6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.729+0000 7f35abfff640 0 client.15732 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:51.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.729+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 shutdown_connections 2024-04-22T06:37:51.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.729+0000 7f35d5768740 1 -- 172.21.15.105:0/3188367192 wait complete. 2024-04-22T06:37:51.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.729+0000 7f35d5768740 20 client.15732 trim_cache size 0 max 16384 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 Processor -- start 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 -- start start 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 0x55b67c803420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c7ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b67c814bd0 con 0x55b67c7fb160 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b67c8152a0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d5768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b67c815970 con 0x55b67c7f4a60 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41098/0 (socket says 172.21.15.105:41098) 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 -- 172.21.15.105:0/2834612921 learned_addr learned my addr 172.21.15.105:0/2834612921 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35cbfff640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c7ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d083c640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 0x55b67c803420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 -- 172.21.15.105:0/2834612921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 msgr2=0x55b67c803420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 0x55b67c803420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 -- 172.21.15.105:0/2834612921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 msgr2=0x55b67c7ff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c7ff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 -- 172.21.15.105:0/2834612921 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b67c7f57f0 con 0x55b67c7fb160 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35cbfff640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c7ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d083c640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 0x55b67c803420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.747+0000 7f35d103d640 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f35c40cec70 tx=0x7f35c40bd050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3266f49bc0ef4018 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35b67fc640 1 -- 172.21.15.105:0/2834612921 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c409ca40 con 0x55b67c7fb160 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35b67fc640 1 -- 172.21.15.105:0/2834612921 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c40b6070 con 0x55b67c7fb160 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35b67fc640 1 -- 172.21.15.105:0/2834612921 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c409d130 con 0x55b67c7fb160 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 -- 172.21.15.105:0/2834612921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 msgr2=0x55b67c801240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f35c40cec70 tx=0x7f35c40bd050 comp rx=0 tx=0).stop 2024-04-22T06:37:51.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 -- 172.21.15.105:0/2834612921 shutdown_connections 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fb160 0x55b67c801240 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7fa7b0 0x55b67c7ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 --2- 172.21.15.105:0/2834612921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7f4a60 0x55b67c803420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 -- 172.21.15.105:0/2834612921 >> 172.21.15.105:0/2834612921 conn(0x55b67c7fb8b0 msgr2=0x55b67c7ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 -- 172.21.15.105:0/2834612921 shutdown_connections 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.748+0000 7f35d5768740 1 -- 172.21.15.105:0/2834612921 wait complete. 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 -- start start 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 Processor -- start 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7f4a60 0x55b67c81f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c818b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b67c816710 con 0x55b67c7fa7b0 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b67c816040 con 0x55b67c7f4a60 2024-04-22T06:37:51.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b67c816de0 con 0x55b67c7fb160 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d5768740 10 client.0 _ll_register_callbacks cb 0x7f35c4098fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35cbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35cbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41110/0 (socket says 172.21.15.105:41110) 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35cbfff640 1 -- 172.21.15.105:0/962046864 learned_addr learned my addr 172.21.15.105:0/962046864 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d083c640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7f4a60 0x55b67c81f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.749+0000 7f35d103d640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c818b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 -- 172.21.15.105:0/962046864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 msgr2=0x55b67c818b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c818b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 -- 172.21.15.105:0/962046864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7f4a60 msgr2=0x55b67c81f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7f4a60 0x55b67c81f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 -- 172.21.15.105:0/962046864 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b67c81cbd0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d103d640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c818b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35cbfff640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009900 tx=0x7f35b802c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6768f7be1b1f893d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:51.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8005030 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f35b803a040 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c40b6070 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f35b80051d0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 10 client.15738 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b67c87b4f0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35c40b6070 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f35b80035f0 con 0x55b67c7fa7b0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 10 client.15738 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 10 client.15738 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.750+0000 7f35d5768740 20 client.15738 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c803ce0 0x55b67c87eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 4 client.15738 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "status"}) -- 0x55b67c7663a0 con 0x55b67c803ce0 2024-04-22T06:37:51.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d083c640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c803ce0 0x55b67c87eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35abfff640 10 client.15738 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d083c640 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c803ce0 0x55b67c87eed0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=263534a8c5541cb2 server_cookie=1357d46b8916bf11 in_seq=0 out_seq=0 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35abfff640 1 -- 172.21.15.105:0/962046864 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b67c7663a0 con 0x55b67c803ce0 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35abfff640 10 client.15738 handle_command_reply: tid=0 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 client.15738 shutdown 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 msgr2=0x55b67c818410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009900 tx=0x7f35b802c9f0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 shutdown_connections 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x55b67c803ce0 0x55b67c87eed0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b67c7fb160 0x55b67c818b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b67c7fa7b0 0x55b67c818410 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35abfff640 0 client.15738 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 --2- 172.21.15.105:0/962046864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b67c7f4a60 0x55b67c81f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.751+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 >> 172.21.15.105:0/962046864 conn(0x55b67c7fb8b0 msgr2=0x55b67c7ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 shutdown_connections 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d5768740 1 -- 172.21.15.105:0/962046864 wait complete. 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d5768740 20 client.15738 trim_cache size 0 max 16384 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 4, 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 14640, 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 125.753694729, 2024-04-22T06:37:51.758 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 27, 2024-04-22T06:37:51.759 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 92, 2024-04-22T06:37:51.759 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 91, 2024-04-22T06:37:51.759 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 139.34366155000001 2024-04-22T06:37:51.759 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:51.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 msgr2=0x7f35bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 0x7f35bc0a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005d00 tx=0x7f35c000b410 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 msgr2=0x7f35cc198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f35c400eb10 tx=0x7f35c400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 shutdown_connections 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f35bc0a3e40 0x7f35bc0a6300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f35cc198730 0x7f35cc198be0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f35cc1072e0 0x7f35cc19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 --2- 172.21.15.105:0/2582629410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35cc102210 0x7f35cc19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 >> 172.21.15.105:0/2582629410 conn(0x7f35cc0fdab0 msgr2=0x7f35cc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 shutdown_connections 2024-04-22T06:37:51.760 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.752+0000 7f35d2aa0640 1 -- 172.21.15.105:0/2582629410 wait complete. 2024-04-22T06:37:51.763 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 -- 172.21.15.105:0/4219383812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec102180 msgr2=0x7f49ec105ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/4219383812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec102180 0x7f49ec105ca0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009900 tx=0x7f49dc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 -- 172.21.15.105:0/4219383812 shutdown_connections 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/4219383812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec106380 0x7f49ec10df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/4219383812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec102180 0x7f49ec105ca0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/4219383812 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec1017d0 0x7f49ec101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 -- 172.21.15.105:0/4219383812 >> 172.21.15.105:0/4219383812 conn(0x7f49ec0773e0 msgr2=0x7f49ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 -- 172.21.15.105:0/4219383812 shutdown_connections 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.863+0000 7f49f0c3a640 1 -- 172.21.15.105:0/4219383812 wait complete. 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 Processor -- start 2024-04-22T06:37:51.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 -- start start 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec106380 0x7f49ec19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec1976e0 0x7f49ec197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1112c0 con 0x7f49ec1017d0 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec111140 con 0x7f49ec1976e0 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49f0c3a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec111440 con 0x7f49ec106380 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41126/0 (socket says 172.21.15.105:41126) 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 -- 172.21.15.105:0/1376713870 learned_addr learned my addr 172.21.15.105:0/1376713870 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ead9d640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec1976e0 0x7f49ec197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec106380 0x7f49ec19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 -- 172.21.15.105:0/1376713870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec106380 msgr2=0x7f49ec19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec106380 0x7f49ec19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 -- 172.21.15.105:0/1376713870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec1976e0 msgr2=0x7f49ec197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec1976e0 0x7f49ec197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.864+0000 7f49ea59c640 1 -- 172.21.15.105:0/1376713870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1982e0 con 0x7f49ec1017d0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49ea59c640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f49e000d770 tx=0x7f49e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49db7fe640 1 -- 172.21.15.105:0/1376713870 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0017070 con 0x7f49ec1017d0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49db7fe640 1 -- 172.21.15.105:0/1376713870 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e00054d0 con 0x7f49ec1017d0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1a43c0 con 0x7f49ec1017d0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49db7fe640 1 -- 172.21.15.105:0/1376713870 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0004c30 con 0x7f49ec1017d0 2024-04-22T06:37:51.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.865+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1a4770 con 0x7f49ec1017d0 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.867+0000 7f49db7fe640 1 -- 172.21.15.105:0/1376713870 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f49e0004e10 con 0x7f49ec1017d0 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.867+0000 7f49f381f740 1 Processor -- start 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 -- start start 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49db7fe640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 0x7f49d40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3620 0x5642787c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.874 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3fd0 0x5642787ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564278727b10 con 0x5642787caed0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5642786c5240 con 0x5642787c3fd0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49f381f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564278723ad0 con 0x5642787c3620 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 0x7f49d40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49db7fe640 1 -- 172.21.15.105:0/1376713870 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f49e00d0490 con 0x7f49ec1017d0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41130/0 (socket says 172.21.15.105:41130) 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 -- 172.21.15.105:0/3627560142 learned_addr learned my addr 172.21.15.105:0/3627560142 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ea59c640 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3620 0x5642787c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3fd0 0x5642787ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 -- 172.21.15.105:0/3627560142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3620 msgr2=0x5642787c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3620 0x5642787c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 -- 172.21.15.105:0/3627560142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3fd0 msgr2=0x5642787ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3fd0 0x5642787ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 -- 172.21.15.105:0/3627560142 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56427843a2d0 con 0x5642787caed0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 0x7f49d40a6380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49e4014be0 tx=0x7f49e400a830 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.868+0000 7f49ead9d640 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f49e4011e00 tx=0x7f49e401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc5b701dbcd4ed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49d8ff9640 1 -- 172.21.15.105:0/3627560142 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e400ede0 con 0x5642787caed0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49d8ff9640 1 -- 172.21.15.105:0/3627560142 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e4003e20 con 0x5642787caed0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49d8ff9640 1 -- 172.21.15.105:0/3627560142 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e40044c0 con 0x5642787caed0 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 -- 172.21.15.105:0/3627560142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 msgr2=0x5642787cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f49e4011e00 tx=0x7f49e401b000 comp rx=0 tx=0).stop 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 -- 172.21.15.105:0/3627560142 shutdown_connections 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787caed0 0x5642787cb350 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3fd0 0x5642787ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 --2- 172.21.15.105:0/3627560142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3620 0x5642787c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 -- 172.21.15.105:0/3627560142 >> 172.21.15.105:0/3627560142 conn(0x5642787bda60 msgr2=0x5642787bde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 -- 172.21.15.105:0/3627560142 shutdown_connections 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.869+0000 7f49f381f740 1 -- 172.21.15.105:0/3627560142 wait complete. 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 -- start start 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 Processor -- start 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3620 0x5642787ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787caed0 0x5642787cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5642787d1e10 con 0x5642787c3fd0 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5642786b0190 con 0x5642787c3620 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5642787d1f90 con 0x5642787caed0 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49f381f740 10 client.0 _ll_register_callbacks cb 0x5642786ab8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41142/0 (socket says 172.21.15.105:41142) 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 -- 172.21.15.105:0/3411349137 learned_addr learned my addr 172.21.15.105:0/3411349137 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49ea59c640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3620 0x5642787ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49ead9d640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787caed0 0x5642787cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 -- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787caed0 msgr2=0x5642787cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787caed0 0x5642787cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 -- 172.21.15.105:0/3411349137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3620 msgr2=0x5642787ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3620 0x5642787ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.870+0000 7f49e9d9b640 1 -- 172.21.15.105:0/3411349137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642785046b0 con 0x5642787c3fd0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f49dc005ab0 tx=0x7f49dc02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbec32777057655 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49dc005230 con 0x5642787c3fd0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f49dc0053d0 con 0x5642787c3fd0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49e4003e20 con 0x5642787c3fd0 2024-04-22T06:37:51.878 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49dc0055d0 con 0x5642787c3fd0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 10 client.15750 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5642787de0e0 con 0x5642787c3fd0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e4003e20 con 0x5642787c3fd0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49dc051050 con 0x5642787c3fd0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 10 client.15750 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 10 client.15750 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 20 client.15750 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5642787e0760 0x5642787e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 4 client.15750 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.871+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564278735530 con 0x5642787e0760 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.872+0000 7f49ea59c640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5642787e0760 0x5642787e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.872+0000 7f49d1ffb640 10 client.15750 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.872+0000 7f49ea59c640 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5642787e0760 0x5642787e1bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=884cb247256834e0 server_cookie=ff5fe8bd65d6175d in_seq=0 out_seq=0 2024-04-22T06:37:51.879 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49d1ffb640 1 -- 172.21.15.105:0/3411349137 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564278735530 con 0x5642787e0760 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49d1ffb640 10 client.15750 handle_command_reply: tid=0 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 client.15750 shutdown 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 msgr2=0x5642787cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f49dc005ab0 tx=0x7f49dc02c9f0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 shutdown_connections 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x5642787e0760 0x5642787e1bd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787caed0 0x5642787cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49d1ffb640 0 client.15750 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3fd0 0x5642787cee40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 --2- 172.21.15.105:0/3411349137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787c3620 0x5642787ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 >> 172.21.15.105:0/3411349137 conn(0x5642787bda60 msgr2=0x5642787bfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 shutdown_connections 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 1 -- 172.21.15.105:0/3411349137 wait complete. 2024-04-22T06:37:51.880 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.873+0000 7f49f381f740 20 client.15750 trim_cache size 0 max 16384 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.890+0000 7f49f381f740 1 Processor -- start 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 -- start start 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5642787dff10 con 0x5642787c3620 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5642787e05e0 con 0x5642787caed0 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49f381f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5642787e0cb0 con 0x5642787c3fd0 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41154/0 (socket says 172.21.15.105:41154) 2024-04-22T06:37:51.896 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 -- 172.21.15.105:0/4229631984 learned_addr learned my addr 172.21.15.105:0/4229631984 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ead9d640 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 -- 172.21.15.105:0/4229631984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 msgr2=0x5642787cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 -- 172.21.15.105:0/4229631984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 msgr2=0x5642787cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 -- 172.21.15.105:0/4229631984 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642787c66c0 con 0x5642787c3620 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49ea59c640 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f49e009bf50 tx=0x7f49e008a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8970100812e530 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49d3fff640 1 -- 172.21.15.105:0/4229631984 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0020040 con 0x5642787c3620 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49d3fff640 1 -- 172.21.15.105:0/4229631984 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e00a0600 con 0x5642787c3620 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.891+0000 7f49d3fff640 1 -- 172.21.15.105:0/4229631984 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e00988c0 con 0x5642787c3620 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- 172.21.15.105:0/4229631984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 msgr2=0x5642787d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f49e009bf50 tx=0x7f49e008a040 comp rx=0 tx=0).stop 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- 172.21.15.105:0/4229631984 shutdown_connections 2024-04-22T06:37:51.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 --2- 172.21.15.105:0/4229631984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787d1030 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- 172.21.15.105:0/4229631984 >> 172.21.15.105:0/4229631984 conn(0x5642787bda60 msgr2=0x5642787c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- 172.21.15.105:0/4229631984 shutdown_connections 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- 172.21.15.105:0/4229631984 wait complete. 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 -- start start 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 Processor -- start 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.892+0000 7f49f381f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5642787e1a50 con 0x5642787c3620 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5642787e1380 con 0x5642787caed0 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5642787e2120 con 0x5642787c3fd0 2024-04-22T06:37:51.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 10 client.0 _ll_register_callbacks cb 0x7f49dc004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41160/0 (socket says 172.21.15.105:41160) 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 -- 172.21.15.105:0/2130078406 learned_addr learned my addr 172.21.15.105:0/2130078406 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ead9d640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 -- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 msgr2=0x5642787ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 -- 172.21.15.105:0/2130078406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 msgr2=0x5642787cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 -- 172.21.15.105:0/2130078406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642788478d0 con 0x5642787c3620 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49ea59c640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f49e000df30 tx=0x7f49e009c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ee9f2ca86f2614 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:51.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e0020050 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f49e008a0b0 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49e00a0600 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f49e008a2b0 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 10 client.15756 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564278848bb0 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.893+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49e00a0600 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f49e00a4a30 con 0x5642787c3620 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 10 client.15756 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 10 client.15756 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 20 client.15756 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56427884d190 0x56427884bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 4 client.15756 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "status"}) -- 0x56427872f3a0 con 0x56427884d190 2024-04-22T06:37:51.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56427884d190 0x56427884bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49e9d9b640 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56427884d190 0x56427884bea0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e6bbdce2d4597fc8 server_cookie=2d0ab3d5ff1f2f5c in_seq=0 out_seq=0 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49d1ffb640 10 client.15756 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49d1ffb640 1 -- 172.21.15.105:0/2130078406 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56427872f3a0 con 0x56427884d190 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49d1ffb640 10 client.15756 handle_command_reply: tid=0 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.894+0000 7f49f381f740 1 client.15756 shutdown 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 msgr2=0x5642787ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f49e000df30 tx=0x7f49e009c150 comp rx=0 tx=0).stop 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 shutdown_connections 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x56427884d190 0x56427884bea0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5642787caed0 0x5642787cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49d1ffb640 0 client.15756 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5642787c3fd0 0x5642787ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 --2- 172.21.15.105:0/2130078406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5642787c3620 0x5642787ce230 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 >> 172.21.15.105:0/2130078406 conn(0x5642787bda60 msgr2=0x5642787cd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 shutdown_connections 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 1 -- 172.21.15.105:0/2130078406 wait complete. 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f381f740 20 client.15756 trim_cache size 0 max 16384 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 3, 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24454, 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:51.902 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 126.860160003, 2024-04-22T06:37:51.903 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 27, 2024-04-22T06:37:51.903 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 92, 2024-04-22T06:37:51.903 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 91, 2024-04-22T06:37:51.903 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 150.379520361 2024-04-22T06:37:51.903 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 msgr2=0x7f49d40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 0x7f49d40a6380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49e4014be0 tx=0x7f49e400a830 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 msgr2=0x7f49ec19d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f49e000d770 tx=0x7f49e000dc30 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 shutdown_connections 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f49d40a3ec0 0x7f49d40a6380 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49ec1976e0 0x7f49ec197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f49ec106380 0x7f49ec19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 --2- 172.21.15.105:0/1376713870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f49ec1017d0 0x7f49ec19d620 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 >> 172.21.15.105:0/1376713870 conn(0x7f49ec0773e0 msgr2=0x7f49ec104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 shutdown_connections 2024-04-22T06:37:51.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:51.895+0000 7f49f0c3a640 1 -- 172.21.15.105:0/1376713870 wait complete. 2024-04-22T06:37:51.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/40132651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef0101a50 msgr2=0x7f4ef0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/40132651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef0101a50 0x7f4ef0105a90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009900 tx=0x7f4ee002bc30 comp rx=0 tx=0).stop 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/40132651 shutdown_connections 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/40132651 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef0106170 0x7f4ef010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/40132651 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef0101a50 0x7f4ef0105a90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/40132651 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef01010a0 0x7f4ef0101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/40132651 >> 172.21.15.105:0/40132651 conn(0x7f4ef0077eb0 msgr2=0x7f4ef00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/40132651 shutdown_connections 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.005+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/40132651 wait complete. 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 Processor -- start 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 -- start start 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef0106170 0x7f4ef0199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 0x7f4ef01a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef0111090 con 0x7f4ef01010a0 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef0110f10 con 0x7f4ef0106170 2024-04-22T06:37:52.008 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4ef4d3b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef0111210 con 0x7f4ef019a030 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41182/0 (socket says 172.21.15.105:41182) 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 -- 172.21.15.105:0/4008335434 learned_addr learned my addr 172.21.15.105:0/4008335434 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef0106170 0x7f4ef0199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 0x7f4ef01a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 -- 172.21.15.105:0/4008335434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 msgr2=0x7f4ef01a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 0x7f4ef01a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 -- 172.21.15.105:0/4008335434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef0106170 msgr2=0x7f4ef0199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef0106170 0x7f4ef0199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 -- 172.21.15.105:0/4008335434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef01a4db0 con 0x7f4ef01010a0 2024-04-22T06:37:52.009 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 0x7f4ef01a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.006+0000 7f4eee59c640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9f0 tx=0x7f4ee400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.007+0000 7f4edf7fe640 1 -- 172.21.15.105:0/4008335434 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400cd80 con 0x7f4ef01010a0 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.007+0000 7f4edf7fe640 1 -- 172.21.15.105:0/4008335434 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee40108e0 con 0x7f4ef01010a0 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.007+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef01a50a0 con 0x7f4ef01010a0 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.007+0000 7f4edf7fe640 1 -- 172.21.15.105:0/4008335434 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4018980 con 0x7f4ef01010a0 2024-04-22T06:37:52.010 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.007+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef01a55c0 con 0x7f4ef01010a0 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4edf7fe640 1 -- 172.21.15.105:0/4008335434 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f4ee4010430 con 0x7f4ef01010a0 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4ef7a05740 1 Processor -- start 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4ef7a05740 1 -- start start 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5be8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.009+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bf4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560eb5b4cb10 con 0x560eb5be9160 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eb5aea240 con 0x560eb5bf1ce0 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560eb5b48ad0 con 0x560eb5be87b0 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41196/0 (socket says 172.21.15.105:41196) 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 -- 172.21.15.105:0/780152358 learned_addr learned my addr 172.21.15.105:0/780152358 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bf4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4edf7fe640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 0x7f4ed80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5be8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 0x7f4ed80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 -- 172.21.15.105:0/780152358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 msgr2=0x560eb5be8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5be8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 -- 172.21.15.105:0/780152358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 msgr2=0x560eb5bf4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bf4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 -- 172.21.15.105:0/780152358 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eb585f2d0 con 0x560eb5be9160 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4edf7fe640 1 -- 172.21.15.105:0/4008335434 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4ee40d0dd0 con 0x7f4ef01010a0 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eee59c640 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ce520 tx=0x7f4ee400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced762a581c098ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 0x7f4ed80a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009370 tx=0x7f4ee8009240 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4edcff9640 1 -- 172.21.15.105:0/780152358 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee409c6d0 con 0x560eb5be9160 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4edcff9640 1 -- 172.21.15.105:0/780152358 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee4010720 con 0x560eb5be9160 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.010+0000 7f4edcff9640 1 -- 172.21.15.105:0/780152358 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40213f0 con 0x560eb5be9160 2024-04-22T06:37:52.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 -- 172.21.15.105:0/780152358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 msgr2=0x560eb5bf1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ce520 tx=0x7f4ee400cf20 comp rx=0 tx=0).stop 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 -- 172.21.15.105:0/780152358 shutdown_connections 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bf4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bf1710 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/780152358 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5be8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 -- 172.21.15.105:0/780152358 >> 172.21.15.105:0/780152358 conn(0x560eb5be2a60 msgr2=0x560eb5be2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 -- 172.21.15.105:0/780152358 shutdown_connections 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.011+0000 7f4ef7a05740 1 -- 172.21.15.105:0/780152358 wait complete. 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 -- start start 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 Processor -- start 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5be87b0 0x560eb5beeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be9160 0x560eb5bef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.021 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560eb5bf4bf0 con 0x560eb5bf1ce0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eb5ad5190 con 0x560eb5be87b0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560eb5bf4d70 con 0x560eb5be9160 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ef7a05740 10 client.0 _ll_register_callbacks cb 0x560eb5ad08b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41208/0 (socket says 172.21.15.105:41208) 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 -- 172.21.15.105:0/2731065964 learned_addr learned my addr 172.21.15.105:0/2731065964 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eee59c640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5be87b0 0x560eb5beeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be9160 0x560eb5bef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 -- 172.21.15.105:0/2731065964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be9160 msgr2=0x560eb5bef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be9160 0x560eb5bef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 -- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5be87b0 msgr2=0x560eb5beeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5be87b0 0x560eb5beeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 -- 172.21.15.105:0/2731065964 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eb59296b0 con 0x560eb5bf1ce0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4eeed9d640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8011e10 tx=0x7f4ee8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658f97a2d61bace4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.012+0000 7f4ed5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8026030 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4ee8011e40 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560eb5b5a530 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ee8030050 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 10 client.15768 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560eb5c00ec0 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560eb5b5a530 con 0x560eb5bf1ce0 2024-04-22T06:37:52.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4ee8022800 con 0x560eb5bf1ce0 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 10 client.15768 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 10 client.15768 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 20 client.15768 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c03550 0x560eb5c049c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 4 client.15768 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eb5b543a0 con 0x560eb5c03550 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.013+0000 7f4eee59c640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c03550 0x560eb5c049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.014+0000 7f4ed5ffb640 10 client.15768 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.014+0000 7f4eee59c640 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c03550 0x560eb5c049c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a593776752bcefcd server_cookie=9df6c87271eaf62c in_seq=0 out_seq=0 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.014+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/2731065964 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560eb5b543a0 con 0x560eb5c03550 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.014+0000 7f4ed5ffb640 10 client.15768 handle_command_reply: tid=0 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 client.15768 shutdown 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 msgr2=0x560eb5bef620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8011e10 tx=0x7f4ee8004a40 comp rx=0 tx=0).stop 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 shutdown_connections 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5bf1ce0 0x560eb5bef620 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c03550 0x560eb5c049c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be9160 0x560eb5bef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ed5ffb640 0 client.15768 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/2731065964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5be87b0 0x560eb5beeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 >> 172.21.15.105:0/2731065964 conn(0x560eb5be2a60 msgr2=0x560eb5be4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 shutdown_connections 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 1 -- 172.21.15.105:0/2731065964 wait complete. 2024-04-22T06:37:52.025 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.015+0000 7f4ef7a05740 20 client.15768 trim_cache size 0 max 16384 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 Processor -- start 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 -- start start 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5bf12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bebe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c02d00 con 0x560eb5be9160 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c033d0 con 0x560eb5bf1ce0 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c03aa0 con 0x560eb5be87b0 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41220/0 (socket says 172.21.15.105:41220) 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 -- 172.21.15.105:0/3949596609 learned_addr learned my addr 172.21.15.105:0/3949596609 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 -- 172.21.15.105:0/3949596609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 msgr2=0x560eb5bf12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5bf12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 -- 172.21.15.105:0/3949596609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 msgr2=0x560eb5bebe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 -- 172.21.15.105:0/3949596609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eb5be99a0 con 0x560eb5be9160 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009900 tx=0x7f4ee00040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84167473ac43b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ed7fff640 1 -- 172.21.15.105:0/3949596609 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0004260 con 0x560eb5be9160 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ed7fff640 1 -- 172.21.15.105:0/3949596609 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee002cc90 con 0x560eb5be9160 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.033+0000 7f4ed7fff640 1 -- 172.21.15.105:0/3949596609 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00048c0 con 0x560eb5be9160 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- 172.21.15.105:0/3949596609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 msgr2=0x560eb5bdd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009900 tx=0x7f4ee00040f0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- 172.21.15.105:0/3949596609 shutdown_connections 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5bebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5bdd800 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/3949596609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5bf12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- 172.21.15.105:0/3949596609 >> 172.21.15.105:0/3949596609 conn(0x560eb5be2a60 msgr2=0x560eb5be2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- 172.21.15.105:0/3949596609 shutdown_connections 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- 172.21.15.105:0/3949596609 wait complete. 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 -- start start 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.034+0000 7f4ef7a05740 1 Processor -- start 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5c0cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5beba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c04840 con 0x560eb5be9160 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c04170 con 0x560eb5bf1ce0 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x560eb5c04f10 con 0x560eb5be87b0 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 10 client.0 _ll_register_callbacks cb 0x7f4ee8009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41224/0 (socket says 172.21.15.105:41224) 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 -- 172.21.15.105:0/465127372 learned_addr learned my addr 172.21.15.105:0/465127372 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 -- 172.21.15.105:0/465127372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 msgr2=0x560eb5c0cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5c0cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 -- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 msgr2=0x560eb5beba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5beba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 -- 172.21.15.105:0/465127372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eb5bec240 con 0x560eb5be9160 2024-04-22T06:37:52.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4eedd9b640 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0036040 tx=0x7f4ee00052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2c2a93348e8922 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ed5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0056060 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4ee002c9e0 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee002cc90 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4ee00046c0 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 10 client.15774 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.035+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560eb5c69500 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ee002cc90 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4ee0059020 con 0x560eb5be9160 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 10 client.15774 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 10 client.15774 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 20 client.15774 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c6dae0 0x560eb5c6c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 4 client.15774 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "status"}) -- 0x560eb5ad0960 con 0x560eb5c6dae0 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4eee59c640 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c6dae0 0x560eb5c6c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ed5ffb640 10 client.15774 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4eee59c640 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c6dae0 0x560eb5c6c7f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a8d54faa75700128 server_cookie=b894f67c15337e5e in_seq=0 out_seq=0 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ed5ffb640 1 -- 172.21.15.105:0/465127372 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560eb5ad0960 con 0x560eb5c6dae0 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ed5ffb640 10 client.15774 handle_command_reply: tid=0 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.036+0000 7f4ef7a05740 1 client.15774 shutdown 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 msgr2=0x560eb5beb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0036040 tx=0x7f4ee00052b0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 shutdown_connections 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560eb5bf1ce0 0x560eb5beba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560eb5be9160 0x560eb5beb4d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x560eb5c6dae0 0x560eb5c6c7f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 --2- 172.21.15.105:0/465127372 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560eb5be87b0 0x560eb5c0cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ed5ffb640 0 client.15774 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 >> 172.21.15.105:0/465127372 conn(0x560eb5be2a60 msgr2=0x560eb5bdd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 shutdown_connections 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 1 -- 172.21.15.105:0/465127372 wait complete. 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef7a05740 20 client.15774 trim_cache size 0 max 16384 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 2, 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24461, 2024-04-22T06:37:52.044 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 128.01337012900001, 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 27, 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 92, 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 91, 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 147.790837369 2024-04-22T06:37:52.045 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 msgr2=0x7f4ed80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 0x7f4ed80a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009370 tx=0x7f4ee8009240 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 msgr2=0x7f4ef0199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9f0 tx=0x7f4ee400eeb0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 shutdown_connections 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4ed80a3e40 0x7f4ed80a6300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ef019a030 0x7f4ef01a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ef0106170 0x7f4ef0199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 --2- 172.21.15.105:0/4008335434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ef01010a0 0x7f4ef0199510 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 >> 172.21.15.105:0/4008335434 conn(0x7f4ef0077eb0 msgr2=0x7f4ef00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 shutdown_connections 2024-04-22T06:37:52.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.037+0000 7f4ef4d3b640 1 -- 172.21.15.105:0/4008335434 wait complete. 2024-04-22T06:37:52.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-22T06:37:52.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 -- 172.21.15.105:0/1235129087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101ad0 msgr2=0x7f06a410dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/1235129087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101ad0 0x7f06a410dcf0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f069c009ca0 tx=0x7f069c02c140 comp rx=0 tx=0).stop 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 -- 172.21.15.105:0/1235129087 shutdown_connections 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/1235129087 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101ad0 0x7f06a410dcf0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/1235129087 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101130 0x7f06a4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/1235129087 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 0x7f06a4106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 -- 172.21.15.105:0/1235129087 >> 172.21.15.105:0/1235129087 conn(0x7f06a4077e30 msgr2=0x7f06a4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 -- 172.21.15.105:0/1235129087 shutdown_connections 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.147+0000 7f06a9fdb640 1 -- 172.21.15.105:0/1235129087 wait complete. 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 Processor -- start 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 -- start start 2024-04-22T06:37:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101ad0 0x7f06a4198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 0x7f06a419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f06a41111f0 con 0x7f06a4101130 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f06a4111070 con 0x7f06a4105ea0 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a9fdb640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f06a4111370 con 0x7f06a4101ad0 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41232/0 (socket says 172.21.15.105:41232) 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 -- 172.21.15.105:0/3104949921 learned_addr learned my addr 172.21.15.105:0/3104949921 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a3fff640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 0x7f06a419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101ad0 0x7f06a4198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 -- 172.21.15.105:0/3104949921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101ad0 msgr2=0x7f06a4198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101ad0 0x7f06a4198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 -- 172.21.15.105:0/3104949921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 msgr2=0x7f06a419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 0x7f06a419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.148+0000 7f06a37fe640 1 -- 172.21.15.105:0/3104949921 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a419d740 con 0x7f06a4101130 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a37fe640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f069400c970 tx=0x7f069400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a0ff9640 1 -- 172.21.15.105:0/3104949921 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694007bb0 con 0x7f06a4101130 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a0ff9640 1 -- 172.21.15.105:0/3104949921 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0694007d50 con 0x7f06a4101130 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a0ff9640 1 -- 172.21.15.105:0/3104949921 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069400f450 con 0x7f06a4101130 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a419da30 con 0x7f06a4101130 2024-04-22T06:37:52.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.149+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a41a52c0 con 0x7f06a4101130 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06a0ff9640 1 -- 172.21.15.105:0/3104949921 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f0694004f20 con 0x7f06a4101130 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 Processor -- start 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 -- start start 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06a0ff9640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 0x7f068c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beb1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82beb1ce0 0x55b82beb4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b82be0cb10 con 0x55b82bea87b0 2024-04-22T06:37:52.157 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b82bdaa240 con 0x55b82bea9160 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06acbc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b82be08ad0 con 0x55b82beb1ce0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06a37fe640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 0x7f068c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.151+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41248/0 (socket says 172.21.15.105:41248) 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 -- 172.21.15.105:0/2614055092 learned_addr learned my addr 172.21.15.105:0/2614055092 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a0ff9640 1 -- 172.21.15.105:0/3104949921 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f06940d07a0 con 0x7f06a4101130 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beb1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82beb1ce0 0x55b82beb4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 -- 172.21.15.105:0/2614055092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82beb1ce0 msgr2=0x55b82beb4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82beb1ce0 0x55b82beb4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 -- 172.21.15.105:0/2614055092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 msgr2=0x55b82beb1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beb1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 -- 172.21.15.105:0/2614055092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b82bb1f2d0 con 0x55b82bea87b0 2024-04-22T06:37:52.158 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a37fe640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 0x7f068c0a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06940cde40 tx=0x7f069401baf0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f06a3fff640 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f069c009ca0 tx=0x7f069c009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11159f6600d198a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f0689ffb640 1 -- 172.21.15.105:0/2614055092 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c048020 con 0x55b82bea87b0 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f0689ffb640 1 -- 172.21.15.105:0/2614055092 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f069c03a040 con 0x55b82bea87b0 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.152+0000 7f0689ffb640 1 -- 172.21.15.105:0/2614055092 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c039050 con 0x55b82bea87b0 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- 172.21.15.105:0/2614055092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 msgr2=0x55b82bea8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f069c009ca0 tx=0x7f069c009960 comp rx=0 tx=0).stop 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- 172.21.15.105:0/2614055092 shutdown_connections 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82beb1ce0 0x55b82beb4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beb1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 --2- 172.21.15.105:0/2614055092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea87b0 0x55b82bea8b90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- 172.21.15.105:0/2614055092 >> 172.21.15.105:0/2614055092 conn(0x55b82bea2a60 msgr2=0x55b82bea2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- 172.21.15.105:0/2614055092 shutdown_connections 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- 172.21.15.105:0/2614055092 wait complete. 2024-04-22T06:37:52.159 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 -- start start 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.153+0000 7f06acbc5740 1 Processor -- start 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82beaec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beaf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b82beb4bf0 con 0x55b82beb1ce0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b82bd95190 con 0x55b82bea9160 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b82beb4d70 con 0x55b82bea87b0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 10 client.0 _ll_register_callbacks cb 0x55b82bd908b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41264/0 (socket says 172.21.15.105:41264) 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 -- 172.21.15.105:0/1582673914 learned_addr learned my addr 172.21.15.105:0/1582673914 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82beaec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beaf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.160 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 -- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 msgr2=0x55b82beaec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82beaec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 -- 172.21.15.105:0/1582673914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 msgr2=0x55b82beaf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beaf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 -- 172.21.15.105:0/1582673914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b82bbe96b0 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06a3fff640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f069c02c680 tx=0x7f069c004210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d305c632f4fd7e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f0682ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c039070 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f069c003d80 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f069c03a040 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f069c048040 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 10 client.15786 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.154+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b82bec0e90 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f069c03a040 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f069c04d040 con 0x55b82beb1ce0 2024-04-22T06:37:52.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 10 client.15786 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 10 client.15786 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 20 client.15786 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bec3000 0x55b82bec4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 4 client.15786 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b82be1a530 con 0x55b82bec3000 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.155+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bec3000 0x55b82bec4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.156+0000 7f0682ffd640 10 client.15786 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.156+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bec3000 0x55b82bec4470 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d19cc121ce4e18a server_cookie=e933a2a1552926e1 in_seq=0 out_seq=0 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.156+0000 7f0682ffd640 1 -- 172.21.15.105:0/1582673914 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b82be1a530 con 0x55b82bec3000 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.156+0000 7f0682ffd640 10 client.15786 handle_command_reply: tid=0 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.156+0000 7f06acbc5740 1 client.15786 shutdown 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 msgr2=0x55b82beaf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f069c02c680 tx=0x7f069c004210 comp rx=0 tx=0).stop 2024-04-22T06:37:52.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 shutdown_connections 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bec3000 0x55b82bec4470 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82beaf6a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea9160 0x55b82beaf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f0682ffd640 0 client.15786 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 --2- 172.21.15.105:0/1582673914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82beaec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 >> 172.21.15.105:0/1582673914 conn(0x55b82bea2a60 msgr2=0x55b82bea4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 shutdown_connections 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 1 -- 172.21.15.105:0/1582673914 wait complete. 2024-04-22T06:37:52.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.157+0000 7f06acbc5740 20 client.15786 trim_cache size 0 max 16384 2024-04-22T06:37:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:52 smithi105 ceph-mon[27916]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:52 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3994201047' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:52.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:52 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2390683649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:52.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.175+0000 7f06acbc5740 1 Processor -- start 2024-04-22T06:37:52.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.175+0000 7f06acbc5740 1 -- start start 2024-04-22T06:37:52.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.175+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea87b0 0x55b82beb1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.175+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 0x55b82bea9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06acbc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec27b0 con 0x55b82beb1ce0 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06acbc5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec2e80 con 0x55b82bea87b0 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06acbc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec3550 con 0x55b82bea9160 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41268/0 (socket says 172.21.15.105:41268) 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 -- 172.21.15.105:0/3067480804 learned_addr learned my addr 172.21.15.105:0/3067480804 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea87b0 0x55b82beb1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 0x55b82bea9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 -- 172.21.15.105:0/3067480804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 msgr2=0x55b82bea9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 0x55b82bea9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 -- 172.21.15.105:0/3067480804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea87b0 msgr2=0x55b82beb1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea87b0 0x55b82beb1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 -- 172.21.15.105:0/3067480804 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b82bea8f00 con 0x55b82beb1ce0 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 0x55b82bea9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06a3fff640 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f069c002230 tx=0x7f069c032e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11ea8d2c3443d66 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06897fa640 1 -- 172.21.15.105:0/3067480804 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c039040 con 0x55b82beb1ce0 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06897fa640 1 -- 172.21.15.105:0/3067480804 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f069c004370 con 0x55b82beb1ce0 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.176+0000 7f06897fa640 1 -- 172.21.15.105:0/3067480804 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c050820 con 0x55b82beb1ce0 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- 172.21.15.105:0/3067480804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 msgr2=0x55b82bea1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f069c002230 tx=0x7f069c032e00 comp rx=0 tx=0).stop 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- 172.21.15.105:0/3067480804 shutdown_connections 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82beb1ce0 0x55b82bea1a00 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea9160 0x55b82bea9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 --2- 172.21.15.105:0/3067480804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82bea87b0 0x55b82beb1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- 172.21.15.105:0/3067480804 >> 172.21.15.105:0/3067480804 conn(0x55b82bea2a60 msgr2=0x55b82be9d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- 172.21.15.105:0/3067480804 shutdown_connections 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- 172.21.15.105:0/3067480804 wait complete. 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.177+0000 7f06acbc5740 1 -- start start 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 Processor -- start 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82becc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82beb1ce0 0x55b82bec6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec42f0 con 0x55b82bea9160 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec3c20 con 0x55b82beb1ce0 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55b82bec49c0 con 0x55b82bea87b0 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 10 client.0 _ll_register_callbacks cb 0x7f069800c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41280/0 (socket says 172.21.15.105:41280) 2024-04-22T06:37:52.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 -- 172.21.15.105:0/4078972837 learned_addr learned my addr 172.21.15.105:0/4078972837 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a3fff640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82becc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a3fff640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82beb1ce0 0x55b82bec6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 -- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 msgr2=0x55b82becc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82becc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 -- 172.21.15.105:0/4078972837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82beb1ce0 msgr2=0x55b82bec6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82beb1ce0 0x55b82bec6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 -- 172.21.15.105:0/4078972837 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b82bec6990 con 0x55b82bea9160 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06a2ffd640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f069800c760 tx=0x7f069800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a619b5592b3dae4b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f0682ffd640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698022060 con 0x55b82bea9160 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f069801c430 con 0x55b82bea9160 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.178+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f069c004370 con 0x55b82bea9160 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f069801c630 con 0x55b82bea9160 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 10 client.15792 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b82bf28fb0 con 0x55b82bea9160 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f069c004370 con 0x55b82bea9160 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0698025030 con 0x55b82bea9160 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 10 client.15792 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 10 client.15792 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 20 client.15792 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bf2dab0 0x55b82bf2c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 4 client.15792 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "status"}) -- 0x55b82be143a0 con 0x55b82bf2dab0 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.179+0000 7f06a3fff640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bf2dab0 0x55b82bf2c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f0682ffd640 10 client.15792 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06a3fff640 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bf2dab0 0x55b82bf2c7c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82169be32c508999 server_cookie=6d9a208c0d36c523 in_seq=0 out_seq=0 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f0682ffd640 1 -- 172.21.15.105:0/4078972837 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b82be143a0 con 0x55b82bf2dab0 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f0682ffd640 10 client.15792 handle_command_reply: tid=0 2024-04-22T06:37:52.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 client.15792 shutdown 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 msgr2=0x55b82bec5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f069800c760 tx=0x7f069800cf30 comp rx=0 tx=0).stop 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 shutdown_connections 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55b82bf2dab0 0x55b82bf2c7c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b82beb1ce0 0x55b82bec6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f0682ffd640 0 client.15792 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b82bea9160 0x55b82bec5d10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 --2- 172.21.15.105:0/4078972837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55b82bea87b0 0x55b82becc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 >> 172.21.15.105:0/4078972837 conn(0x55b82bea2a60 msgr2=0x55b82bea1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 shutdown_connections 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 1 -- 172.21.15.105:0/4078972837 wait complete. 2024-04-22T06:37:52.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.180+0000 7f06acbc5740 20 client.15792 trim_cache size 0 max 16384 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 1, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24472, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 129.164460854, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 27, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 92, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 91, 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 145.31946250300001 2024-04-22T06:37:52.187 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:52.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 msgr2=0x7f068c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 0x7f068c0a6300 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06940cde40 tx=0x7f069401baf0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 msgr2=0x7f06a4198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f069400c970 tx=0x7f069400ce30 comp rx=0 tx=0).stop 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 shutdown_connections 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f068c0a3e40 0x7f068c0a6300 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06a4105ea0 0x7f06a419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f06a4101ad0 0x7f06a4198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 --2- 172.21.15.105:0/3104949921 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06a4101130 0x7f06a4198750 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 >> 172.21.15.105:0/3104949921 conn(0x7f06a4077e30 msgr2=0x7f06a4104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 shutdown_connections 2024-04-22T06:37:52.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.181+0000 7f06a9fdb640 1 -- 172.21.15.105:0/3104949921 wait complete. 2024-04-22T06:37:52.191 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-22T06:37:52.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 -- 172.21.15.105:0/753196218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c102190 msgr2=0x7f929c102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 --2- 172.21.15.105:0/753196218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c102190 0x7f929c102570 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9294009ca0 tx=0x7f929402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 -- 172.21.15.105:0/753196218 shutdown_connections 2024-04-22T06:37:52.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 --2- 172.21.15.105:0/753196218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 0x7f929c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.292 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 --2- 172.21.15.105:0/753196218 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102b40 0x7f929c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 --2- 172.21.15.105:0/753196218 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c102190 0x7f929c102570 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 -- 172.21.15.105:0/753196218 >> 172.21.15.105:0/753196218 conn(0x7f929c0793a0 msgr2=0x7f929c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 -- 172.21.15.105:0/753196218 shutdown_connections 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 -- 172.21.15.105:0/753196218 wait complete. 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.291+0000 7f92a1367640 1 Processor -- start 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 -- start start 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102190 0x7f929c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 0x7f929c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f929c1121c0 con 0x7f929c198640 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112040 con 0x7f929c107260 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f92a1367640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112340 con 0x7f929c102190 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.293 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41302/0 (socket says 172.21.15.105:41302) 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 -- 172.21.15.105:0/333077511 learned_addr learned my addr 172.21.15.105:0/333077511 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929a7fc640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 0x7f929c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102190 0x7f929c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 -- 172.21.15.105:0/333077511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102190 msgr2=0x7f929c19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102190 0x7f929c19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 -- 172.21.15.105:0/333077511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 msgr2=0x7f929c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 0x7f929c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 -- 172.21.15.105:0/333077511 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c199160 con 0x7f929c198640 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f929b7fe640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f928c00d950 tx=0x7f928c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.292+0000 7f9283fff640 1 -- 172.21.15.105:0/333077511 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c017030 con 0x7f929c198640 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.293+0000 7f9283fff640 1 -- 172.21.15.105:0/333077511 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f928c004020 con 0x7f929c198640 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.293+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c199450 con 0x7f929c198640 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.293+0000 7f9283fff640 1 -- 172.21.15.105:0/333077511 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c003b50 con 0x7f929c198640 2024-04-22T06:37:52.294 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.293+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c1a5600 con 0x7f929c198640 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f9283fff640 1 -- 172.21.15.105:0/333077511 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7f928c01a070 con 0x7f929c198640 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 Processor -- start 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 -- start start 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e47b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e50ce0 0x5604c5e53130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5604c5dabb10 con 0x5604c5e48160 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604c5d49240 con 0x5604c5e50ce0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f92a4038740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604c5da7ad0 con 0x5604c5e477b0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f9283fff640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 0x7f92840a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f929affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.295+0000 7f929affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41304/0 (socket says 172.21.15.105:41304) 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 -- 172.21.15.105:0/4228029664 learned_addr learned my addr 172.21.15.105:0/4228029664 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929b7fe640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 0x7f92840a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929a7fc640 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e47b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f9283fff640 1 -- 172.21.15.105:0/333077511 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f928c0d0150 con 0x7f929c198640 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929a7fc640 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e50ce0 0x5604c5e53130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 -- 172.21.15.105:0/4228029664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 msgr2=0x5604c5e47b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e47b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 -- 172.21.15.105:0/4228029664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e50ce0 msgr2=0x5604c5e53130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e50ce0 0x5604c5e53130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 -- 172.21.15.105:0/4228029664 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c5abe2d0 con 0x5604c5e48160 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929affd640 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f929402c700 tx=0x7f9294032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7356f4c5d665d1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f929b7fe640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 0x7f92840a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f928c0cd790 tx=0x7f928c00dfa0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f92817fa640 1 -- 172.21.15.105:0/4228029664 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294043070 con 0x5604c5e48160 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f92817fa640 1 -- 172.21.15.105:0/4228029664 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9294032970 con 0x5604c5e48160 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.296+0000 7f92817fa640 1 -- 172.21.15.105:0/4228029664 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9294039040 con 0x5604c5e48160 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- 172.21.15.105:0/4228029664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 msgr2=0x5604c5e50710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f929402c700 tx=0x7f9294032340 comp rx=0 tx=0).stop 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- 172.21.15.105:0/4228029664 shutdown_connections 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e50ce0 0x5604c5e53130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e48160 0x5604c5e50710 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- 172.21.15.105:0/4228029664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e47b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- 172.21.15.105:0/4228029664 >> 172.21.15.105:0/4228029664 conn(0x5604c5e41a60 msgr2=0x5604c5e41e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- 172.21.15.105:0/4228029664 shutdown_connections 2024-04-22T06:37:52.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- 172.21.15.105:0/4228029664 wait complete. 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- start start 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 Processor -- start 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e53bf0 con 0x5604c5e50ce0 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604c5d34190 con 0x5604c5e48160 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.297+0000 7f92a4038740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e53d70 con 0x5604c5e477b0 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f92a4038740 10 client.0 _ll_register_callbacks cb 0x5604c5d2f8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41314/0 (socket says 172.21.15.105:41314) 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 -- 172.21.15.105:0/1013688532 learned_addr learned my addr 172.21.15.105:0/1013688532 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929affd640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.304 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 -- 172.21.15.105:0/1013688532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 msgr2=0x5604c5e4dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 -- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 msgr2=0x5604c5e4e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 -- 172.21.15.105:0/1013688532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c5b886b0 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f92900109a0 tx=0x7f92900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2bb5623676542c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f927eff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000fb50 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f929000fcf0 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9294032970 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9290005020 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f92a4038740 10 client.15804 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5604c5e5fed0 con 0x5604c5e50ce0 2024-04-22T06:37:52.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.298+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9294032970 con 0x5604c5e50ce0 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9290026020 con 0x5604c5e50ce0 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 10 client.15804 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 10 client.15804 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 20 client.15804 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5e638b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 4 client.15804 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604c5db9530 con 0x5604c5e62440 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f929affd640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5e638b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f929affd640 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5e638b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcbb6523f2f07dc8 server_cookie=d4b338a331cd746e in_seq=0 out_seq=0 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.299+0000 7f927eff5640 10 client.15804 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f927eff5640 1 -- 172.21.15.105:0/1013688532 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604c5db9530 con 0x5604c5e62440 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f927eff5640 10 client.15804 handle_command_reply: tid=0 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 client.15804 shutdown 2024-04-22T06:37:52.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 msgr2=0x5604c5e4e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f92900109a0 tx=0x7f92900046c0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 shutdown_connections 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5e638b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4e6b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f927eff5640 0 client.15804 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 --2- 172.21.15.105:0/1013688532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 >> 172.21.15.105:0/1013688532 conn(0x5604c5e41a60 msgr2=0x5604c5e43bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 shutdown_connections 2024-04-22T06:37:52.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 1 -- 172.21.15.105:0/1013688532 wait complete. 2024-04-22T06:37:52.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.300+0000 7f92a4038740 20 client.15804 trim_cache size 0 max 16384 2024-04-22T06:37:52.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 Processor -- start 2024-04-22T06:37:52.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 -- start start 2024-04-22T06:37:52.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e50360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e48590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e61bf0 con 0x5604c5e50ce0 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e622c0 con 0x5604c5e48160 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f92a4038740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e62990 con 0x5604c5e477b0 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41324/0 (socket says 172.21.15.105:41324) 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 -- 172.21.15.105:0/2858936344 learned_addr learned my addr 172.21.15.105:0/2858936344 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 -- 172.21.15.105:0/2858936344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 msgr2=0x5604c5e50360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e50360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 -- 172.21.15.105:0/2858936344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 msgr2=0x5604c5e48590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e48590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.318+0000 7f929a7fc640 1 -- 172.21.15.105:0/2858936344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604c5e47f00 con 0x5604c5e50ce0 2024-04-22T06:37:52.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f929a7fc640 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9290009380 tx=0x7f92900104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73996d9aeda9646 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f9280ff9640 1 -- 172.21.15.105:0/2858936344 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000f790 con 0x5604c5e50ce0 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f9280ff9640 1 -- 172.21.15.105:0/2858936344 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92900285e0 con 0x5604c5e50ce0 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f9280ff9640 1 -- 172.21.15.105:0/2858936344 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290005220 con 0x5604c5e50ce0 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 -- 172.21.15.105:0/2858936344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 msgr2=0x5604c5e6bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9290009380 tx=0x7f92900104b0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.333 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 -- 172.21.15.105:0/2858936344 shutdown_connections 2024-04-22T06:37:52.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e6bf80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e48590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 --2- 172.21.15.105:0/2858936344 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e50360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 -- 172.21.15.105:0/2858936344 >> 172.21.15.105:0/2858936344 conn(0x5604c5e41a60 msgr2=0x5604c5e3c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 -- 172.21.15.105:0/2858936344 shutdown_connections 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.319+0000 7f92a4038740 1 -- 172.21.15.105:0/2858936344 wait complete. 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 -- start start 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 Processor -- start 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e63730 con 0x5604c5e50ce0 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e63060 con 0x5604c5e48160 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5604c5e63e00 con 0x5604c5e477b0 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f92a4038740 10 client.0 _ll_register_callbacks cb 0x7f9290005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:41334/0 (socket says 172.21.15.105:41334) 2024-04-22T06:37:52.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 -- 172.21.15.105:0/1012293244 learned_addr learned my addr 172.21.15.105:0/1012293244 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:52.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929affd640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 -- 172.21.15.105:0/1012293244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 msgr2=0x5604c5e4b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 -- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 msgr2=0x5604c5e4d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 -- 172.21.15.105:0/1012293244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604c5e4e460 con 0x5604c5e50ce0 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f929a7fc640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f92900109a0 tx=0x7f9290018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1e3c9d23c842b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.320+0000 7f927eff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929003a020 con 0x5604c5e50ce0 2024-04-22T06:37:52.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f9290004050 con 0x5604c5e50ce0 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92900285e0 con 0x5604c5e50ce0 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9290004250 con 0x5604c5e50ce0 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 10 client.15810 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5604c5ecc2d0 con 0x5604c5e50ce0 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92900285e0 con 0x5604c5e50ce0 2024-04-22T06:37:52.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9290031020 con 0x5604c5e50ce0 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 10 client.15810 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 10 client.15810 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 20 client.15810 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5ecf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 4 client.15810 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-22T06:37:52.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "status"}) -- 0x5604c5db33a0 con 0x5604c5e62440 2024-04-22T06:37:52.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.321+0000 7f929affd640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5ecf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:52.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f927eff5640 10 client.15810 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:52.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f929affd640 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5ecf620 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c272573ab7b37e5a server_cookie=47275ec636e206d1 in_seq=0 out_seq=0 2024-04-22T06:37:52.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f927eff5640 1 -- 172.21.15.105:0/1012293244 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5604c5db33a0 con 0x5604c5e62440 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f927eff5640 10 client.15810 handle_command_reply: tid=0 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 client.15810 shutdown 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 msgr2=0x5604c5e4ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f92900109a0 tx=0x7f9290018040 comp rx=0 tx=0).stop 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 shutdown_connections 2024-04-22T06:37:52.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5604c5e62440 0x5604c5ecf620 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5604c5e50ce0 0x5604c5e4ddf0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5604c5e48160 0x5604c5e4d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f927eff5640 0 client.15810 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 --2- 172.21.15.105:0/1012293244 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5604c5e477b0 0x5604c5e4b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 >> 172.21.15.105:0/1012293244 conn(0x5604c5e41a60 msgr2=0x5604c5e4c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 shutdown_connections 2024-04-22T06:37:52.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 1 -- 172.21.15.105:0/1012293244 wait complete. 2024-04-22T06:37:52.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.322+0000 7f92a4038740 20 client.15810 trim_cache size 0 max 16384 2024-04-22T06:37:52.344 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_fsid": "48924e24-0071-11ef-bc93-c7b262605968", 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": 0, 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "id": 24500, 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "want_state": "up:active", 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "state": "up:active", 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "fs_name": "cephfs", 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "rank_uptime": 131.519138118, 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "mdsmap_epoch": 27, 2024-04-22T06:37:52.345 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch": 92, 2024-04-22T06:37:52.346 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_epoch_barrier": 91, 2024-04-22T06:37:52.346 INFO:teuthology.orchestra.run.smithi105.stdout: "uptime": 137.039371665 2024-04-22T06:37:52.346 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 msgr2=0x7f92840a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 0x7f92840a6380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f928c0cd790 tx=0x7f928c00dfa0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 msgr2=0x7f929c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f928c00d950 tx=0x7f928c00de10 comp rx=0 tx=0).stop 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 shutdown_connections 2024-04-22T06:37:52.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f92840a3ec0 0x7f92840a6380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f929c198640 0x7f929c198af0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f929c107260 0x7f929c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 --2- 172.21.15.105:0/333077511 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f929c102190 0x7f929c19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 >> 172.21.15.105:0/333077511 conn(0x7f929c0793a0 msgr2=0x7f929c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 shutdown_connections 2024-04-22T06:37:52.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:37:52.323+0000 7f92a1367640 1 -- 172.21.15.105:0/333077511 wait complete. 2024-04-22T06:37:52.349 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-22T06:37:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:52 smithi112 ceph-mon[31236]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:52 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3994201047' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:52.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:52 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2390683649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:52.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:52 smithi177 ceph-mon[31346]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:52.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:52 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3994201047' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:37:52.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:52 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2390683649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:37:54.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:54 smithi105 ceph-mon[27916]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:54 smithi112 ceph-mon[31236]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:54 smithi177 ceph-mon[31346]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-22T06:37:56.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:56 smithi105 ceph-mon[27916]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-22T06:37:56.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:56 smithi112 ceph-mon[31236]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-22T06:37:56.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:56 smithi177 ceph-mon[31346]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 56 op/s 2024-04-22T06:37:58.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:37:58 smithi105 ceph-mon[27916]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:37:58.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:37:58 smithi112 ceph-mon[31236]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:37:58.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:37:58 smithi177 ceph-mon[31346]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-04-22T06:37:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.655+0000 7fc403385640 1 -- 172.21.15.105:0/2558848736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc100fb0 msgr2=0x7fc3fc110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.655+0000 7fc403385640 1 --2- 172.21.15.105:0/2558848736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc110be0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009ca0 tx=0x7fc3f402c180 comp rx=0 tx=0).stop 2024-04-22T06:37:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- 172.21.15.105:0/2558848736 shutdown_connections 2024-04-22T06:37:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- 172.21.15.105:0/2558848736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc110be0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- 172.21.15.105:0/2558848736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc103250 0x7fc3fc100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- 172.21.15.105:0/2558848736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc102e70 0x7fc3fc100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- 172.21.15.105:0/2558848736 >> 172.21.15.105:0/2558848736 conn(0x7fc3fc077390 msgr2=0x7fc3fc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- 172.21.15.105:0/2558848736 shutdown_connections 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- 172.21.15.105:0/2558848736 wait complete. 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 Processor -- start 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- start start 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc1a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 0x7fc3fc19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1143b0 con 0x7fc3fc102e70 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc114230 con 0x7fc3fc103250 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.656+0000 7fc403385640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc114530 con 0x7fc3fc100fb0 2024-04-22T06:37:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34076/0 (socket says 172.21.15.105:34076) 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 -- 172.21.15.105:0/2892268675 learned_addr learned my addr 172.21.15.105:0/2892268675 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc401922640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 0x7fc3fc19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc401121640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc1a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 -- 172.21.15.105:0/2892268675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc100fb0 msgr2=0x7fc3fc1a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc1a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 -- 172.21.15.105:0/2892268675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 msgr2=0x7fc3fc19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 0x7fc3fc19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 -- 172.21.15.105:0/2892268675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3fc19aef0 con 0x7fc3fc102e70 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc401922640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 0x7fc3fc19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc400920640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00e9f0 tx=0x7fc3ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc3f27fc640 1 -- 172.21.15.105:0/2892268675 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00cd80 con 0x7fc3fc102e70 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc3f27fc640 1 -- 172.21.15.105:0/2892268675 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec0054d0 con 0x7fc3fc102e70 2024-04-22T06:37:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3fc19b1e0 con 0x7fc3fc102e70 2024-04-22T06:37:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc3f27fc640 1 -- 172.21.15.105:0/2892268675 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec010690 con 0x7fc3fc102e70 2024-04-22T06:37:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.657+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3fc1a74f0 con 0x7fc3fc102e70 2024-04-22T06:37:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc3f27fc640 1 -- 172.21.15.105:0/2892268675 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 138874+0+0 (secure 0 0 0) 0x7fc3ec004e90 con 0x7fc3fc102e70 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 Processor -- start 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 -- start start 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecef96fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefa24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557eceefb8d0 con 0x557ecef978d0 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ecee98e80 con 0x557ecef96b60 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.660+0000 7fc405f75740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557eceef7890 con 0x557ecefa00e0 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401121640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecef96fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34090/0 (socket says 172.21.15.105:34090) 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 -- 172.21.15.105:0/2584921026 learned_addr learned my addr 172.21.15.105:0/2584921026 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc3f27fc640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 0x7fc3e80a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc400920640 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefa24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 -- 172.21.15.105:0/2584921026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 msgr2=0x557ecefa24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefa24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 -- 172.21.15.105:0/2584921026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 msgr2=0x557ecef96fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecef96fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 -- 172.21.15.105:0/2584921026 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557eceeba320 con 0x557ecef978d0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401121640 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecef96fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc3f27fc640 1 -- 172.21.15.105:0/2892268675 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc3ec0d0f00 con 0x7fc3fc102e70 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 0x7fc3e80a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc401922640 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009ca0 tx=0x7fc3f40324b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac2979eefd9f3e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc3e77fe640 1 -- 172.21.15.105:0/2584921026 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4039040 con 0x557ecef978d0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc3e77fe640 1 -- 172.21.15.105:0/2584921026 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4004710 con 0x557ecef978d0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.661+0000 7fc3e77fe640 1 -- 172.21.15.105:0/2584921026 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4034640 con 0x557ecef978d0 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 -- 172.21.15.105:0/2584921026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 msgr2=0x557ecef96620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc401922640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 0x7fc3e80a63a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc3f404b000 tx=0x7fc3f4044000 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009ca0 tx=0x7fc3f40324b0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 -- 172.21.15.105:0/2584921026 shutdown_connections 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefa24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecef96fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 --2- 172.21.15.105:0/2584921026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef96620 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 -- 172.21.15.105:0/2584921026 >> 172.21.15.105:0/2584921026 conn(0x557ecef5dc90 msgr2=0x557ecef91850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 -- 172.21.15.105:0/2584921026 shutdown_connections 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.662+0000 7fc405f75740 1 -- 172.21.15.105:0/2584921026 wait complete. 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 -- start start 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 Processor -- start 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef978d0 0x557ecef9cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecef9d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557ecefa2e10 con 0x557ecef96b60 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ecee83eb0 con 0x557ecef978d0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ecefa2f90 con 0x557ecefa00e0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc405f75740 10 client.0 _ll_register_callbacks cb 0x557eceeb0b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc401121640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc401121640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34100/0 (socket says 172.21.15.105:34100) 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc401121640 1 -- 172.21.15.105:0/1900771230 learned_addr learned my addr 172.21.15.105:0/1900771230 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc401922640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecef9d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.663+0000 7fc400920640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef978d0 0x557ecef9cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 -- 172.21.15.105:0/1900771230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 msgr2=0x557ecef9d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecef9d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 -- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef978d0 msgr2=0x557ecef9cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef978d0 0x557ecef9cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 -- 172.21.15.105:0/1900771230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ecec0e210 con 0x557ecef96b60 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc401121640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80126b0 tx=0x7fc3f8012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9342ae7eb0938ea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8004260 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc3f8004400 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3f4004710 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3f80119a0 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 10 client.15822 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ececda620 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4004710 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc3f8024020 con 0x557ecef96b60 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 10 client.15822 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 20 client.15822 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecefb00c0 0x557ecefb24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 4 client.15822 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:37:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.664+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ecef0f880 con 0x557ecefb00c0 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.665+0000 7fc400920640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecefb00c0 0x557ecefb24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.665+0000 7fc3e4ff9640 10 client.15822 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.665+0000 7fc400920640 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecefb00c0 0x557ecefb24d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355b2735916a6ffe server_cookie=fac958f606bd6c7d in_seq=0 out_seq=0 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1900771230 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ecef0f880 con 0x557ecefb00c0 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc3e4ff9640 10 client.15822 handle_command_reply: tid=0 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 client.15822 shutdown 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 msgr2=0x557ecef9fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80126b0 tx=0x7fc3f8012b70 comp rx=0 tx=0).stop 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 shutdown_connections 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecefb00c0 0x557ecefb24d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecef9d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef978d0 0x557ecef9cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 --2- 172.21.15.105:0/1900771230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef96b60 0x557ecef9fa50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc3e4ff9640 0 client.15822 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 >> 172.21.15.105:0/1900771230 conn(0x557ecef5dc90 msgr2=0x557ecef935a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 shutdown_connections 2024-04-22T06:37:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 1 -- 172.21.15.105:0/1900771230 wait complete. 2024-04-22T06:37:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.666+0000 7fc405f75740 20 client.15822 trim_cache size 0 max 16384 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 Processor -- start 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 -- start start 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecefabd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefb8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1afb30 con 0x557ecef978d0 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1b1670 con 0x557ecef96b60 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc405f75740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1b0200 con 0x557ecefa00e0 2024-04-22T06:37:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34112/0 (socket says 172.21.15.105:34112) 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.681+0000 7fc400920640 1 -- 172.21.15.105:0/181945769 learned_addr learned my addr 172.21.15.105:0/181945769 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc401121640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecefabd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc401922640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefb8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 -- 172.21.15.105:0/181945769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 msgr2=0x557ecefb8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefb8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 -- 172.21.15.105:0/181945769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 msgr2=0x557ecefabd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecefabd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 -- 172.21.15.105:0/181945769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ecef8cf30 con 0x557ecef978d0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc401121640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecefabd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc401922640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefb8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc400920640 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec099690 tx=0x7fc3ec0bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b90d79cd6357160 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc3e6ffd640 1 -- 172.21.15.105:0/181945769 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec005680 con 0x557ecef978d0 2024-04-22T06:37:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc3e6ffd640 1 -- 172.21.15.105:0/181945769 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec09d3c0 con 0x557ecef978d0 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.682+0000 7fc3e6ffd640 1 -- 172.21.15.105:0/181945769 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec09b090 con 0x557ecef978d0 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- 172.21.15.105:0/181945769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 msgr2=0x557ecefb7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec099690 tx=0x7fc3ec0bd040 comp rx=0 tx=0).stop 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- 172.21.15.105:0/181945769 shutdown_connections 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecefa00e0 0x557ecefb8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecefb7de0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- 172.21.15.105:0/181945769 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecef96b60 0x557ecefabd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- 172.21.15.105:0/181945769 >> 172.21.15.105:0/181945769 conn(0x557ecef5dc90 msgr2=0x557ecef97fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- 172.21.15.105:0/181945769 shutdown_connections 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- 172.21.15.105:0/181945769 wait complete. 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- start start 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 Processor -- start 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecef96b60 0x557ecef99ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecefa00e0 0x557ecef9a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1b0fa0 con 0x557ecef978d0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1b08d0 con 0x557ecefa00e0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc1adb10 con 0x557ecef96b60 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.683+0000 7fc405f75740 10 client.0 _ll_register_callbacks cb 0x7fc3f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34120/0 (socket says 172.21.15.105:34120) 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 -- 172.21.15.105:0/1827970012 learned_addr learned my addr 172.21.15.105:0/1827970012 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc401121640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecef96b60 0x557ecef99ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 -- 172.21.15.105:0/1827970012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecef96b60 msgr2=0x557ecef99ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecef96b60 0x557ecef99ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 -- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecefa00e0 msgr2=0x557ecef9a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecefa00e0 0x557ecef9a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 -- 172.21.15.105:0/1827970012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ecf015720 con 0x557ecef978d0 2024-04-22T06:37:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc400920640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec09cc50 tx=0x7fc3ec005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9194bd3735cb76 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc3e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec066060 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc3ec0046c0 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3ec09d3c0 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec0a1070 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc405f75740 10 client.15828 fetch_fsmap learned FSMap version 27 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ecf0169d0 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.684+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3ec09d3c0 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc3ec0bd0b0 con 0x557ecef978d0 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 10 client.15828 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 20 client.15828 populate_metadata read hostname 'smithi105' 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecf0170d0 0x557ecf0194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 4 client.15828 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:37:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ecee9b3d0 con 0x557ecf0170d0 2024-04-22T06:37:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc401121640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecf0170d0 0x557ecf0194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:37:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc3e4ff9640 10 client.15828 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.685+0000 7fc401121640 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecf0170d0 0x557ecf0194f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3fd828a6117b36a server_cookie=c5bcc86c43248e89 in_seq=0 out_seq=0 2024-04-22T06:37:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc3e4ff9640 1 -- 172.21.15.105:0/1827970012 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ecee9b3d0 con 0x557ecf0170d0 2024-04-22T06:37:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc3e4ff9640 10 client.15828 handle_command_reply: tid=0 2024-04-22T06:37:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 client.15828 shutdown 2024-04-22T06:37:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 msgr2=0x557ecef9a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec09cc50 tx=0x7fc3ec005710 comp rx=0 tx=0).stop 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 shutdown_connections 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ecf0170d0 0x557ecf0194f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ecefa00e0 0x557ecef9a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ecef978d0 0x557ecef9a0e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 --2- 172.21.15.105:0/1827970012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ecef96b60 0x557ecef99ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc3e4ff9640 0 client.15828 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 >> 172.21.15.105:0/1827970012 conn(0x557ecef5dc90 msgr2=0x557ecef9b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.696+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 shutdown_connections 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc405f75740 1 -- 172.21.15.105:0/1827970012 wait complete. 2024-04-22T06:37:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc405f75740 20 client.15828 trim_cache size 0 max 16384 2024-04-22T06:37:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:37:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:37:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:37:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 msgr2=0x7fc3e80a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 0x7fc3e80a63a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc3f404b000 tx=0x7fc3f4044000 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 msgr2=0x7fc3fc1a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00e9f0 tx=0x7fc3ec00edf0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 shutdown_connections 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc3e80a3ee0 0x7fc3e80a63a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc3fc103250 0x7fc3fc19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3fc102e70 0x7fc3fc1a0ab0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 --2- 172.21.15.105:0/2892268675 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc3fc100fb0 0x7fc3fc1a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 >> 172.21.15.105:0/2892268675 conn(0x7fc3fc077390 msgr2=0x7fc3fc0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 shutdown_connections 2024-04-22T06:37:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:37:58.697+0000 7fc403385640 1 -- 172.21.15.105:0/2892268675 wait complete. 2024-04-22T06:38:00.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-mon[27916]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-22T06:38:00.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:00 smithi112 ceph-mon[31236]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-22T06:38:00.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:00 smithi177 ceph-mon[31346]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-22T06:38:00.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:38:00.752+0000 7fb00329b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:38:00.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:38:00.752+0000 7fb00329b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:38:00.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:38:00.752+0000 7fb00329b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:38:00.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:38:00.752+0000 7fb00329b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:38:00.918 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:38:00 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:38:00.752+0000 7fb00329b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-22T06:38:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:02 smithi105 ceph-mon[27916]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T06:38:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:02 smithi105 ceph-mon[27916]: mgrmap e25: x(active, since 24s), standbys: y, z 2024-04-22T06:38:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:02 smithi112 ceph-mon[31236]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T06:38:02.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:02 smithi112 ceph-mon[31236]: mgrmap e25: x(active, since 24s), standbys: y, z 2024-04-22T06:38:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:02 smithi177 ceph-mon[31346]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-22T06:38:02.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:02 smithi177 ceph-mon[31346]: mgrmap e25: x(active, since 24s), standbys: y, z 2024-04-22T06:38:04.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:04 smithi105 ceph-mon[27916]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T06:38:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:04 smithi112 ceph-mon[31236]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T06:38:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:04 smithi177 ceph-mon[31346]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-22T06:38:06.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:06 smithi105 ceph-mon[27916]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:06.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:06 smithi112 ceph-mon[31236]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:06.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:06 smithi177 ceph-mon[31346]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:07.350 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 16.6 2024-04-22T06:38:07.351 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:38:07.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 -- 172.21.15.105:0/1695601101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 msgr2=0x7f224810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 --2- 172.21.15.105:0/1695601101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f224810ee30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2240009ca0 tx=0x7f224002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 -- 172.21.15.105:0/1695601101 shutdown_connections 2024-04-22T06:38:07.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 --2- 172.21.15.105:0/1695601101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f224810ee30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 --2- 172.21.15.105:0/1695601101 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2248102b40 0x7f2248106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 --2- 172.21.15.105:0/1695601101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 0x7f2248102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 -- 172.21.15.105:0/1695601101 >> 172.21.15.105:0/1695601101 conn(0x7f22480793e0 msgr2=0x7f22480797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 -- 172.21.15.105:0/1695601101 shutdown_connections 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.464+0000 7f224f25c640 1 -- 172.21.15.105:0/1695601101 wait complete. 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 Processor -- start 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 -- start start 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 0x7f22481986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f224819cfb0 0x7f224819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f22481121b0 con 0x7f2248107260 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2248112030 con 0x7f2248102190 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224f25c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2248112330 con 0x7f224819cfb0 2024-04-22T06:38:07.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60944/0 (socket says 172.21.15.105:60944) 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 -- 172.21.15.105:0/3558130375 learned_addr learned my addr 172.21.15.105:0/3558130375 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224cff8640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 0x7f22481986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f224d7f9640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f224819cfb0 0x7f224819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 -- 172.21.15.105:0/3558130375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f224819cfb0 msgr2=0x7f224819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f224819cfb0 0x7f224819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 -- 172.21.15.105:0/3558130375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 msgr2=0x7f22481986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 0x7f22481986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.465+0000 7f2247fff640 1 -- 172.21.15.105:0/3558130375 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224819daa0 con 0x7f2248107260 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f2247fff640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2234005fd0 tx=0x7f223400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234017070 con 0x7f2248107260 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2234009c30 con 0x7f2248107260 2024-04-22T06:38:07.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f224819dd90 con 0x7f2248107260 2024-04-22T06:38:07.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234003d30 con 0x7f2248107260 2024-04-22T06:38:07.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.466+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22481a5580 con 0x7f2248107260 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.467+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2248069230 con 0x7f2248107260 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.468+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f2234003ed0 con 0x7f2248107260 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.469+0000 7f2245ffb640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 0x7f22380a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.469+0000 7f224cff8640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 0x7f22380a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.469+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f22340d0650 con 0x7f2248107260 2024-04-22T06:38:07.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.469+0000 7f224cff8640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 0x7f22380a6390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bbe0 tx=0x7f223c00b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.474+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2234020030 con 0x7f2248107260 2024-04-22T06:38:07.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.688+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2248102b40 con 0x7f2248107260 2024-04-22T06:38:07.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.689+0000 7f2245ffb640 1 -- 172.21.15.105:0/3558130375 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f223409cb20 con 0x7f2248107260 2024-04-22T06:38:07.688 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:38:07.688 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:38:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 msgr2=0x7f22380a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 0x7f22380a6390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f223c00bbe0 tx=0x7f223c00b390 comp rx=0 tx=0).stop 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 msgr2=0x7f2248198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2234005fd0 tx=0x7f223400dde0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 shutdown_connections 2024-04-22T06:38:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f22380a3ed0 0x7f22380a6390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f224819cfb0 0x7f224819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2248107260 0x7f2248198c30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 --2- 172.21.15.105:0/3558130375 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2248102190 0x7f22481986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.691+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 >> 172.21.15.105:0/3558130375 conn(0x7f22480793e0 msgr2=0x7f22480ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.692+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 shutdown_connections 2024-04-22T06:38:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.692+0000 7f224f25c640 1 -- 172.21.15.105:0/3558130375 wait complete. 2024-04-22T06:38:07.701 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 60 --expiration 76.6 --await' 2024-04-22T06:38:07.701 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db / --timeout 60 --expiration 76.6 --await 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 -- 172.21.15.105:0/2324139934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 msgr2=0x7f423810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 --2- 172.21.15.105:0/2324139934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f423810ee30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4230009ca0 tx=0x7f423002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 -- 172.21.15.105:0/2324139934 shutdown_connections 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 --2- 172.21.15.105:0/2324139934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f423810ee30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 --2- 172.21.15.105:0/2324139934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102b40 0x7f4238106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 --2- 172.21.15.105:0/2324139934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4238102190 0x7f4238102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.810+0000 7f423f470640 1 -- 172.21.15.105:0/2324139934 >> 172.21.15.105:0/2324139934 conn(0x7f4238079310 msgr2=0x7f4238079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- 172.21.15.105:0/2324139934 shutdown_connections 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- 172.21.15.105:0/2324139934 wait complete. 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 Processor -- start 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- start start 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102190 0x7f4238198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 0x7f423819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f42381121b0 con 0x7f4238107260 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4238112030 con 0x7f423819cf30 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423f470640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4238112330 con 0x7f4238102190 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60960/0 (socket says 172.21.15.105:60960) 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.811+0000 7f423ca0b640 1 -- 172.21.15.105:0/278532677 learned_addr learned my addr 172.21.15.105:0/278532677 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423da0d640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 0x7f423819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423d20c640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102190 0x7f4238198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 -- 172.21.15.105:0/278532677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102190 msgr2=0x7f4238198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102190 0x7f4238198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 -- 172.21.15.105:0/278532677 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 msgr2=0x7f423819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 0x7f423819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 -- 172.21.15.105:0/278532677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423819da20 con 0x7f4238107260 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423da0d640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 0x7f423819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423ca0b640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f422800d740 tx=0x7f422800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f422e7fc640 1 -- 172.21.15.105:0/278532677 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228017070 con 0x7f4238107260 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f422e7fc640 1 -- 172.21.15.105:0/278532677 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f422800de50 con 0x7f4238107260 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423819dd10 con 0x7f4238107260 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f422e7fc640 1 -- 172.21.15.105:0/278532677 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228004c10 con 0x7f4238107260 2024-04-22T06:38:07.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.812+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42381a54b0 con 0x7f4238107260 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.814+0000 7f422e7fc640 1 -- 172.21.15.105:0/278532677 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f4228004dd0 con 0x7f4238107260 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 Processor -- start 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 -- start start 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ee9c0 0x55ce207eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207f7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ce2074f0b0 con 0x55ce207f7ec0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ce20700680 con 0x55ce207ef2e0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f424205c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ce2070cc60 con 0x55ce207ee9c0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f422e7fc640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 0x7f42240a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60964/0 (socket says 172.21.15.105:60964) 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 -- 172.21.15.105:0/774682289 learned_addr learned my addr 172.21.15.105:0/774682289 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423da0d640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 0x7f42240a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423d20c640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423da0d640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ee9c0 0x55ce207eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f422e7fc640 1 -- 172.21.15.105:0/278532677 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f42280d05b0 con 0x7f4238107260 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 -- 172.21.15.105:0/774682289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ee9c0 msgr2=0x55ce207eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ee9c0 0x55ce207eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 -- 172.21.15.105:0/774682289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 msgr2=0x55ce207f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423ca0b640 1 -- 172.21.15.105:0/774682289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce20531c90 con 0x55ce207f7ec0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.815+0000 7f423d20c640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f423da0d640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 0x7f42240a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4230009a60 tx=0x7f4230009970 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f423ca0b640 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42280986b0 tx=0x7f422809bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19aaadedc27f2c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f42237fe640 1 -- 172.21.15.105:0/774682289 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228020070 con 0x55ce207f7ec0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f42237fe640 1 -- 172.21.15.105:0/774682289 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42280a4f10 con 0x55ce207f7ec0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f42237fe640 1 -- 172.21.15.105:0/774682289 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42280bf1c0 con 0x55ce207f7ec0 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f424205c740 1 -- 172.21.15.105:0/774682289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 msgr2=0x55ce207fa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f424205c740 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42280986b0 tx=0x7f422809bcc0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f424205c740 1 -- 172.21.15.105:0/774682289 shutdown_connections 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f424205c740 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207fa2b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.816+0000 7f424205c740 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 --2- 172.21.15.105:0/774682289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ee9c0 0x55ce207eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- 172.21.15.105:0/774682289 >> 172.21.15.105:0/774682289 conn(0x55ce207e90f0 msgr2=0x55ce207e94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- 172.21.15.105:0/774682289 shutdown_connections 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- 172.21.15.105:0/774682289 wait complete. 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- start start 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 Processor -- start 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ef2e0 0x55ce207f5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ce20752400 con 0x55ce207f7ec0 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ce2070e840 con 0x55ce207ee9c0 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ce20757720 con 0x55ce207ef2e0 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f424205c740 10 client.0 _ll_register_callbacks cb 0x55ce2075fea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f423da0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f423da0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60966/0 (socket says 172.21.15.105:60966) 2024-04-22T06:38:07.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.817+0000 7f423da0d640 1 -- 172.21.15.105:0/3272602658 learned_addr learned my addr 172.21.15.105:0/3272602658 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423d20c640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423ca0b640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ef2e0 0x55ce207f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 -- 172.21.15.105:0/3272602658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ef2e0 msgr2=0x55ce207f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ef2e0 0x55ce207f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 -- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 msgr2=0x55ce207f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 -- 172.21.15.105:0/3272602658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce20592420 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f423da0d640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4230009ca0 tx=0x7f4230045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555eec11403e8fba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f4220ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230049040 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4230004240 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42280a4f10 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f423005b050 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f424205c740 10 client.15852 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce20425580 con 0x55ce207f7ec0 2024-04-22T06:38:07.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.818+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42280a4f10 con 0x55ce207f7ec0 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f423004c070 con 0x55ce207f7ec0 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 10 client.15852 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 10 client.15852 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 20 client.15852 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce20807c70 0x55ce2080a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 4 client.15852 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce20760b10 con 0x55ce20807c70 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f423d20c640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce20807c70 0x55ce2080a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f423d20c640 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce20807c70 0x55ce2080a0a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5477058fd241a8dd server_cookie=cc7051c61253576d in_seq=0 out_seq=0 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.819+0000 7f4220ff9640 10 client.15852 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f4220ff9640 1 -- 172.21.15.105:0/3272602658 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce20760b10 con 0x55ce20807c70 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f4220ff9640 10 client.15852 handle_command_reply: tid=0 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 client.15852 shutdown 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 msgr2=0x55ce207f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4230009ca0 tx=0x7f4230045070 comp rx=0 tx=0).stop 2024-04-22T06:38:07.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 shutdown_connections 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce20807c70 0x55ce2080a0a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207f7ec0 0x55ce207f59a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207ef2e0 0x55ce207f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f4220ff9640 0 client.15852 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 --2- 172.21.15.105:0/3272602658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 >> 172.21.15.105:0/3272602658 conn(0x55ce207e90f0 msgr2=0x55ce207eae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 shutdown_connections 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.820+0000 7f424205c740 1 -- 172.21.15.105:0/3272602658 wait complete. 2024-04-22T06:38:07.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.821+0000 7f424205c740 20 client.15852 trim_cache size 0 max 16384 2024-04-22T06:38:07.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.837+0000 7f424205c740 1 Processor -- start 2024-04-22T06:38:07.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.837+0000 7f424205c740 1 -- start start 2024-04-22T06:38:07.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.837+0000 7f424205c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.837+0000 7f424205c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f424205c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2080f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f424205c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f42381af3f0 con 0x55ce207ef2e0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f424205c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42381adf80 con 0x55ce207ee9c0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f424205c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42381ae650 con 0x55ce207f7ec0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60968/0 (socket says 172.21.15.105:60968) 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 -- 172.21.15.105:0/602970894 learned_addr learned my addr 172.21.15.105:0/602970894 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423d20c640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423da0d640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2080f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 -- 172.21.15.105:0/602970894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 msgr2=0x55ce2080f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2080f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 -- 172.21.15.105:0/602970894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 msgr2=0x55ce207f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 -- 172.21.15.105:0/602970894 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce207eb3f0 con 0x55ce207ef2e0 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423d20c640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f423ca0b640 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4228009610 tx=0x7f422809c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa0c0dd152c99b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f4222ffd640 1 -- 172.21.15.105:0/602970894 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42280b5070 con 0x55ce207ef2e0 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f4222ffd640 1 -- 172.21.15.105:0/602970894 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42280a0650 con 0x55ce207ef2e0 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.838+0000 7f4222ffd640 1 -- 172.21.15.105:0/602970894 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228020070 con 0x55ce207ef2e0 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- 172.21.15.105:0/602970894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 msgr2=0x55ce20803810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4228009610 tx=0x7f422809c3e0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- 172.21.15.105:0/602970894 shutdown_connections 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2080f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ef2e0 0x55ce20803810 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 --2- 172.21.15.105:0/602970894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ee9c0 0x55ce207f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- 172.21.15.105:0/602970894 >> 172.21.15.105:0/602970894 conn(0x55ce207e90f0 msgr2=0x55ce207f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- 172.21.15.105:0/602970894 shutdown_connections 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- 172.21.15.105:0/602970894 wait complete. 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 -- start start 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.839+0000 7f424205c740 1 Processor -- start 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207fc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2086d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f42381ab910 con 0x55ce207ee9c0 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f42381aed20 con 0x55ce207ef2e0 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f42381abf60 con 0x55ce207f7ec0 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 10 client.0 _ll_register_callbacks cb 0x7f4230009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60982/0 (socket says 172.21.15.105:60982) 2024-04-22T06:38:07.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 -- 172.21.15.105:0/3806392731 learned_addr learned my addr 172.21.15.105:0/3806392731 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423da0d640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2086d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423ca0b640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 -- 172.21.15.105:0/3806392731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 msgr2=0x55ce2086d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2086d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 -- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 msgr2=0x55ce207fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 -- 172.21.15.105:0/3806392731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce2086d7a0 con 0x55ce207ee9c0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423ca0b640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423da0d640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2086d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f423d20c640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f423400b880 tx=0x7f423400f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3ac41e8f5a57b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f4220ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423401c7c0 con 0x55ce207ee9c0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f4234025420 con 0x55ce207ee9c0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42280a0650 con 0x55ce207ee9c0 2024-04-22T06:38:07.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4234025620 con 0x55ce207ee9c0 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 10 client.15858 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.840+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce2086eac0 con 0x55ce207ee9c0 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42280a0650 con 0x55ce207ee9c0 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f423402d050 con 0x55ce207ee9c0 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 10 client.15858 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 10 client.15858 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 20 client.15858 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce2086f260 0x55ce20871680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 4 client.15858 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 76.6, "await": true}] 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 76.6, "await": true}) -- 0x55ce2075a980 con 0x55ce2086f260 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f423ca0b640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce2086f260 0x55ce20871680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f4220ff9640 10 client.15858 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:07.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.841+0000 7f423ca0b640 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce2086f260 0x55ce20871680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=670e6d502c1d3b14 server_cookie=74c0530faa1ab0fb in_seq=0 out_seq=0 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f4220ff9640 1 -- 172.21.15.105:0/3806392731 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55ce2075a980 con 0x55ce2086f260 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f4220ff9640 10 client.15858 handle_command_reply: tid=0 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 client.15858 shutdown 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 msgr2=0x55ce207fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f423400b880 tx=0x7f423400f8b0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 shutdown_connections 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ce2086f260 0x55ce20871680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ce207f7ec0 0x55ce2086d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce207ef2e0 0x55ce207fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 --2- 172.21.15.105:0/3806392731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ce207ee9c0 0x55ce207fbee0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f4220ff9640 0 client.15858 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 >> 172.21.15.105:0/3806392731 conn(0x55ce207e90f0 msgr2=0x55ce207f36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 shutdown_connections 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 1 -- 172.21.15.105:0/3806392731 wait complete. 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.854+0000 7f424205c740 20 client.15858 trim_cache size 0 max 16384 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 27, 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 2, 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:38:07.854 INFO:teuthology.orchestra.run.smithi105.stdout: "2ad732f1": { 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 2, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 76.6, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:07.855 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:07.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:07.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:07.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:07.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:07.856 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 msgr2=0x7f42240a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 0x7f42240a6390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4230009a60 tx=0x7f4230009970 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 msgr2=0x7f4238198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f422800d740 tx=0x7f422800dc00 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 shutdown_connections 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f42240a3ed0 0x7f42240a6390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f423819cf30 0x7f423819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4238107260 0x7f4238198bb0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 --2- 172.21.15.105:0/278532677 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4238102190 0x7f4238198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 >> 172.21.15.105:0/278532677 conn(0x7f4238079310 msgr2=0x7f4238100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 shutdown_connections 2024-04-22T06:38:07.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:07.855+0000 7f423f470640 1 -- 172.21.15.105:0/278532677 wait complete. 2024-04-22T06:38:07.864 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 2ad732f1, quiesce took 0.5 seconds. Will release after: 16.6 2024-04-22T06:38:08.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:08 smithi105 ceph-mon[27916]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:38:08.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:08 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3558130375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:08.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:08 smithi112 ceph-mon[31236]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:38:08.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:08 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3558130375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:08 smithi177 ceph-mon[31346]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-22T06:38:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:08 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3558130375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:10.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:10 smithi105 ceph-mon[27916]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:10.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:10 smithi112 ceph-mon[31236]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:10.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:10 smithi177 ceph-mon[31346]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 -- 172.21.15.105:0/368096086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 msgr2=0x7f7884110d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 --2- 172.21.15.105:0/368096086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f7884110d20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f787c009900 tx=0x7f787c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 -- 172.21.15.105:0/368096086 shutdown_connections 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 --2- 172.21.15.105:0/368096086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f7884110d20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 --2- 172.21.15.105:0/368096086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78841016a0 0x7f788410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 --2- 172.21.15.105:0/368096086 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7884100d80 0x7f7884101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 -- 172.21.15.105:0/368096086 >> 172.21.15.105:0/368096086 conn(0x7f7884077fd0 msgr2=0x7f78840783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 -- 172.21.15.105:0/368096086 shutdown_connections 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 -- 172.21.15.105:0/368096086 wait complete. 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.793+0000 7f7889414640 1 Processor -- start 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 -- start start 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7884100d80 0x7f78841a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78841016a0 0x7f78841a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78841143b0 con 0x7f788410e930 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7884114230 con 0x7f7884100d80 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7889414640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7884114530 con 0x7f78841016a0 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60990/0 (socket says 172.21.15.105:60990) 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 -- 172.21.15.105:0/224857513 learned_addr learned my addr 172.21.15.105:0/224857513 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f7882ffd640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7884100d80 0x7f78841a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78827fc640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78841016a0 0x7f78841a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 -- 172.21.15.105:0/224857513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78841016a0 msgr2=0x7f78841a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78841016a0 0x7f78841a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 -- 172.21.15.105:0/224857513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7884100d80 msgr2=0x7f78841a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7884100d80 0x7f78841a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 -- 172.21.15.105:0/224857513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788419aff0 con 0x7f788410e930 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.794+0000 7f78837fe640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f787c005940 tx=0x7f787c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.795+0000 7f786bfff640 1 -- 172.21.15.105:0/224857513 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c043020 con 0x7f788410e930 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.795+0000 7f786bfff640 1 -- 172.21.15.105:0/224857513 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c02cb70 con 0x7f788410e930 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.795+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788419b280 con 0x7f788410e930 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.795+0000 7f786bfff640 1 -- 172.21.15.105:0/224857513 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c003b50 con 0x7f788410e930 2024-04-22T06:38:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.795+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841a75f0 con 0x7f788410e930 2024-04-22T06:38:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f786bfff640 1 -- 172.21.15.105:0/224857513 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f787c046020 con 0x7f788410e930 2024-04-22T06:38:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 Processor -- start 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 -- start start 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b6dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b770e0 0x563131b794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563131ad28d0 con 0x563131b6e8d0 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563131a6fe80 con 0x563131b770e0 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f788c008740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563131ace890 con 0x563131b6db60 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32774/0 (socket says 172.21.15.105:32774) 2024-04-22T06:38:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.797+0000 7f78827fc640 1 -- 172.21.15.105:0/3325640629 learned_addr learned my addr 172.21.15.105:0/3325640629 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b6dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 -- 172.21.15.105:0/3325640629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 msgr2=0x563131b6dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b6dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 -- 172.21.15.105:0/3325640629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b770e0 msgr2=0x563131b794d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f786bfff640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 0x7f786c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b770e0 0x563131b794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 -- 172.21.15.105:0/3325640629 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563131a91320 con 0x563131b6e8d0 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b6dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78837fe640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 0x7f786c0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78827fc640 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f787800bc90 tx=0x7f7878011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77fc1124abdc4cc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f786bfff640 1 -- 172.21.15.105:0/224857513 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f787c0f2b50 con 0x7f788410e930 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78697fa640 1 -- 172.21.15.105:0/3325640629 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787801a400 con 0x563131b6e8d0 2024-04-22T06:38:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78697fa640 1 -- 172.21.15.105:0/3325640629 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787801aa50 con 0x563131b6e8d0 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.798+0000 7f78697fa640 1 -- 172.21.15.105:0/3325640629 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878012980 con 0x563131b6e8d0 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- 172.21.15.105:0/3325640629 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 msgr2=0x563131b6d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f78837fe640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 0x7f786c0a64b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f787c0f0240 tx=0x7f787c085040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f787800bc90 tx=0x7f7878011750 comp rx=0 tx=0).stop 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- 172.21.15.105:0/3325640629 shutdown_connections 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b770e0 0x563131b794d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b6dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 --2- 172.21.15.105:0/3325640629 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b6d620 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- 172.21.15.105:0/3325640629 >> 172.21.15.105:0/3325640629 conn(0x563131b34c90 msgr2=0x563131b68850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- 172.21.15.105:0/3325640629 shutdown_connections 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- 172.21.15.105:0/3325640629 wait complete. 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 -- start start 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.799+0000 7f788c008740 1 Processor -- start 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b769e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 0x563131b73de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563131b79e10 con 0x563131b770e0 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563131a5aeb0 con 0x563131b6e8d0 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563131b79f90 con 0x563131b6db60 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 10 client.0 _ll_register_callbacks cb 0x563131a87b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32784/0 (socket says 172.21.15.105:32784) 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 -- 172.21.15.105:0/3535527171 learned_addr learned my addr 172.21.15.105:0/3535527171 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b769e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 -- 172.21.15.105:0/3535527171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 msgr2=0x563131b769e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b769e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 -- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 msgr2=0x563131b73de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 0x563131b73de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 0x563131b73de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 -- 172.21.15.105:0/3535527171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631317e5210 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 0x563131b73de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f78827fc640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f787800bf40 tx=0x7f787800b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717a35425696295e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f7866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787801a810 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f7878012700 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.800+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f787801aa50 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7878012900 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 10 client.15870 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5631318b1620 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787801aa50 con 0x563131b770e0 2024-04-22T06:38:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7878028c10 con 0x563131b770e0 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 10 client.15870 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 10 client.15870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 20 client.15870 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131b86f30 0x563131b89320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 4 client.15870 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563131ae6880 con 0x563131b86f30 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.801+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131b86f30 0x563131b89320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.802+0000 7f7882ffd640 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131b86f30 0x563131b89320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38283ed453c9a3d4 server_cookie=a8ec492a3dc05e9a in_seq=0 out_seq=0 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.802+0000 7f7866ff5640 10 client.15870 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.802+0000 7f7866ff5640 1 -- 172.21.15.105:0/3535527171 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563131ae6880 con 0x563131b86f30 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.802+0000 7f7866ff5640 10 client.15870 handle_command_reply: tid=0 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 client.15870 shutdown 2024-04-22T06:38:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 msgr2=0x563131b74340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f787800bf40 tx=0x7f787800b3a0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 shutdown_connections 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131b86f30 0x563131b89320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b74340 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6e8d0 0x563131b73de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f7866ff5640 0 client.15870 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 --2- 172.21.15.105:0/3535527171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6db60 0x563131b769e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 >> 172.21.15.105:0/3535527171 conn(0x563131b34c90 msgr2=0x563131b6a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 shutdown_connections 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 1 -- 172.21.15.105:0/3535527171 wait complete. 2024-04-22T06:38:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.803+0000 7f788c008740 20 client.15870 trim_cache size 0 max 16384 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 Processor -- start 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 -- start start 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b82d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b770e0 0x563131b8ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78841afbc0 con 0x563131b6e8d0 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78841b1700 con 0x563131b6db60 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f788c008740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78841b0290 con 0x563131b770e0 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.823 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32800/0 (socket says 172.21.15.105:32800) 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 -- 172.21.15.105:0/1780032876 learned_addr learned my addr 172.21.15.105:0/1780032876 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f78827fc640 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b82d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 -- 172.21.15.105:0/1780032876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b770e0 msgr2=0x563131b8ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b770e0 0x563131b8ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 -- 172.21.15.105:0/1780032876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 msgr2=0x563131b82d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b82d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f7882ffd640 1 -- 172.21.15.105:0/1780032876 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563131b6ce30 con 0x563131b6e8d0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.819+0000 7f78827fc640 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b82d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.820+0000 7f7882ffd640 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x563131b899b0 tx=0x7f7874009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843fb565d43fb309 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.820+0000 7f7868ff9640 1 -- 172.21.15.105:0/1780032876 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874013d30 con 0x563131b6e8d0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.820+0000 7f7868ff9640 1 -- 172.21.15.105:0/1780032876 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787401e030 con 0x563131b6e8d0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.820+0000 7f7868ff9640 1 -- 172.21.15.105:0/1780032876 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874005770 con 0x563131b6e8d0 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.820+0000 7f788c008740 1 -- 172.21.15.105:0/1780032876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 msgr2=0x563131b73380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x563131b899b0 tx=0x7f7874009680 comp rx=0 tx=0).stop 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- 172.21.15.105:0/1780032876 shutdown_connections 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b770e0 0x563131b8ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b6e8d0 0x563131b73380 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- 172.21.15.105:0/1780032876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b82d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- 172.21.15.105:0/1780032876 >> 172.21.15.105:0/1780032876 conn(0x563131b34c90 msgr2=0x563131b6f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- 172.21.15.105:0/1780032876 shutdown_connections 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- 172.21.15.105:0/1780032876 wait complete. 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- start start 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 Processor -- start 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b7f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6e8d0 0x563131b704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f78841b1030 con 0x563131b770e0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78841b0960 con 0x563131b6db60 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f78841adbe0 con 0x563131b6e8d0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f788c008740 10 client.0 _ll_register_callbacks cb 0x7f787800bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f78827fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32810/0 (socket says 172.21.15.105:32810) 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f78827fc640 1 -- 172.21.15.105:0/974209827 learned_addr learned my addr 172.21.15.105:0/974209827 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f7882ffd640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6e8d0 0x563131b704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.821+0000 7f78827fc640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b7f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 -- 172.21.15.105:0/974209827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6e8d0 msgr2=0x563131b704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6e8d0 0x563131b704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 -- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 msgr2=0x563131b7f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b7f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 -- 172.21.15.105:0/974209827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563131b89780 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f78827fc640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7878001eb0 tx=0x7f7878031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dc1fa529d654a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7878036070 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f7878031a50 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f787401e030 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7878042050 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 10 client.15876 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563131becd20 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787401e030 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f787803b060 con 0x563131b770e0 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 10 client.15876 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.822+0000 7f788c008740 20 client.15876 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131bed4a0 0x563131bef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f788c008740 4 client.15876 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563131a723d0 con 0x563131bed4a0 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f7882ffd640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131bed4a0 0x563131bef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f7866ff5640 10 client.15876 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.823+0000 7f7882ffd640 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131bed4a0 0x563131bef8c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87dfe54b590fff98 server_cookie=c0fa8cc77e7a1e9b in_seq=0 out_seq=0 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f7866ff5640 1 -- 172.21.15.105:0/974209827 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563131a723d0 con 0x563131bed4a0 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f7866ff5640 10 client.15876 handle_command_reply: tid=0 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 client.15876 shutdown 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 msgr2=0x563131b709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7878001eb0 tx=0x7f7878031750 comp rx=0 tx=0).stop 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 shutdown_connections 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563131bed4a0 0x563131bef8c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563131b770e0 0x563131b709e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563131b6e8d0 0x563131b704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f7866ff5640 0 client.15876 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 --2- 172.21.15.105:0/974209827 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563131b6db60 0x563131b7f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 >> 172.21.15.105:0/974209827 conn(0x563131b34c90 msgr2=0x563131b726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 shutdown_connections 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 1 -- 172.21.15.105:0/974209827 wait complete. 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.831+0000 7f788c008740 20 client.15876 trim_cache size 0 max 16384 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:38:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 msgr2=0x7f786c0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 0x7f786c0a64b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f787c0f0240 tx=0x7f787c085040 comp rx=0 tx=0).stop 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 msgr2=0x7f788419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f787c005940 tx=0x7f787c0045e0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 shutdown_connections 2024-04-22T06:38:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f786c0a3ff0 0x7f786c0a64b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f788410e930 0x7f788419a730 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78841016a0 0x7f78841a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 --2- 172.21.15.105:0/224857513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7884100d80 0x7f78841a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 >> 172.21.15.105:0/224857513 conn(0x7f7884077fd0 msgr2=0x7f788410fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 shutdown_connections 2024-04-22T06:38:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:11.832+0000 7f7889414640 1 -- 172.21.15.105:0/224857513 wait complete. 2024-04-22T06:38:12.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:12 smithi105 ceph-mon[27916]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-22T06:38:12.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:12 smithi112 ceph-mon[31236]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-22T06:38:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:12 smithi177 ceph-mon[31346]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-22T06:38:14.220 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:38:14.221 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:38:14.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 -- 172.21.15.105:0/2356644630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac44107240 msgr2=0x7fac4410ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 --2- 172.21.15.105:0/2356644630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac44107240 0x7fac4410ee10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009ca0 tx=0x7fac3c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 -- 172.21.15.105:0/2356644630 shutdown_connections 2024-04-22T06:38:14.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 --2- 172.21.15.105:0/2356644630 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac44107240 0x7fac4410ee10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 --2- 172.21.15.105:0/2356644630 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44102b20 0x7fac44106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 --2- 172.21.15.105:0/2356644630 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 0x7fac44102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 -- 172.21.15.105:0/2356644630 >> 172.21.15.105:0/2356644630 conn(0x7fac440793a0 msgr2=0x7fac440797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 -- 172.21.15.105:0/2356644630 shutdown_connections 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.335+0000 7fac4ae81640 1 -- 172.21.15.105:0/2356644630 wait complete. 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 Processor -- start 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 -- start start 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 0x7fac44198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 0x7fac44198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fac441121b0 con 0x7fac4419cfd0 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fac44112030 con 0x7fac44102170 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4ae81640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac44112330 con 0x7fac44107240 2024-04-22T06:38:14.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32832/0 (socket says 172.21.15.105:32832) 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac4941e640 1 -- 172.21.15.105:0/3042367865 learned_addr learned my addr 172.21.15.105:0/3042367865 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 0x7fac44198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.336+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 0x7fac44198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 -- 172.21.15.105:0/3042367865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 msgr2=0x7fac44198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 0x7fac44198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 -- 172.21.15.105:0/3042367865 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 msgr2=0x7fac44198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 0x7fac44198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 -- 172.21.15.105:0/3042367865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4419dac0 con 0x7fac4419cfd0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 0x7fac44198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4941e640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005e30 tx=0x7fac3c034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac39ffb640 1 -- 172.21.15.105:0/3042367865 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c039040 con 0x7fac4419cfd0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac39ffb640 1 -- 172.21.15.105:0/3042367865 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c044070 con 0x7fac4419cfd0 2024-04-22T06:38:14.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4419dd50 con 0x7fac4419cfd0 2024-04-22T06:38:14.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac39ffb640 1 -- 172.21.15.105:0/3042367865 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c003e60 con 0x7fac4419cfd0 2024-04-22T06:38:14.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.337+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac441a5510 con 0x7fac4419cfd0 2024-04-22T06:38:14.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.339+0000 7fac39ffb640 1 -- 172.21.15.105:0/3042367865 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fac3c0324c0 con 0x7fac4419cfd0 2024-04-22T06:38:14.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 Processor -- start 2024-04-22T06:38:14.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 -- start start 2024-04-22T06:38:14.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887b580 0x56072887b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac39ffb640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 0x7fac2c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728886e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5607287e01e0 con 0x56072887bea0 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56072877d390 con 0x56072887b580 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4da77740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5607287dc1a0 con 0x560728884a80 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 0x7fac2c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32836/0 (socket says 172.21.15.105:32836) 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 -- 172.21.15.105:0/4087712006 learned_addr learned my addr 172.21.15.105:0/4087712006 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac39ffb640 1 -- 172.21.15.105:0/3042367865 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fac3c0f2c00 con 0x7fac4419cfd0 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887b580 0x56072887b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac4941e640 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728886e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 -- 172.21.15.105:0/4087712006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 msgr2=0x560728886e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728886e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 -- 172.21.15.105:0/4087712006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887b580 msgr2=0x56072887b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887b580 0x56072887b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.340+0000 7fac3bfff640 1 -- 172.21.15.105:0/4087712006 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607285bc7d0 con 0x56072887bea0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 0x7fac2c0a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fac4000bbe0 tx=0x7fac4000b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac3bfff640 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fac340135a0 tx=0x7fac3400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484dbafa2d274b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac2affd640 1 -- 172.21.15.105:0/4087712006 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34019030 con 0x56072887bea0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac2affd640 1 -- 172.21.15.105:0/4087712006 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac340054d0 con 0x56072887bea0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac2affd640 1 -- 172.21.15.105:0/4087712006 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac34011720 con 0x56072887bea0 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 -- 172.21.15.105:0/4087712006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 msgr2=0x560728884540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fac340135a0 tx=0x7fac3400b690 comp rx=0 tx=0).stop 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 -- 172.21.15.105:0/4087712006 shutdown_connections 2024-04-22T06:38:14.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728886e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887bea0 0x560728884540 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 --2- 172.21.15.105:0/4087712006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887b580 0x56072887b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.341+0000 7fac4da77740 1 -- 172.21.15.105:0/4087712006 >> 172.21.15.105:0/4087712006 conn(0x560728875cb0 msgr2=0x560728876090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- 172.21.15.105:0/4087712006 shutdown_connections 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- 172.21.15.105:0/4087712006 wait complete. 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- start start 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 Processor -- start 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728881b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728882050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x560728887870 con 0x560728884a80 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5607287992f0 con 0x56072887bea0 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5607288879f0 con 0x56072887b580 2024-04-22T06:38:14.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4da77740 10 client.0 _ll_register_callbacks cb 0x5607287a6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32844/0 (socket says 172.21.15.105:32844) 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.342+0000 7fac4941e640 1 -- 172.21.15.105:0/144567813 learned_addr learned my addr 172.21.15.105:0/144567813 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac3bfff640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728882050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac48c1d640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728881b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 -- 172.21.15.105:0/144567813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 msgr2=0x560728881b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728881b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 -- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 msgr2=0x560728882050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728882050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 -- 172.21.15.105:0/144567813 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607284f23f0 con 0x560728884a80 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac48c1d640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728881b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4941e640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0be7e0 tx=0x7fac3c0be620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6f6477ec9e2c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0d1d80 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fac3c0327b0 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac340054d0 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac3c0d1ba0 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4da77740 10 client.15888 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607284b20f0 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac340054d0 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fac3c03c030 con 0x560728884a80 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4da77740 10 client.15888 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.343+0000 7fac4da77740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac4da77740 20 client.15888 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x560728894a20 0x560728896ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac4da77740 4 client.15888 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:14.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607287edc00 con 0x560728894a20 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac48c1d640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x560728894a20 0x560728896ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac48c1d640 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x560728894a20 0x560728896ea0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=792c4284acd70dfc server_cookie=12fb13bf77af08fa in_seq=0 out_seq=0 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.344+0000 7fac23fff640 10 client.15888 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac23fff640 1 -- 172.21.15.105:0/144567813 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607287edc00 con 0x560728894a20 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac23fff640 10 client.15888 handle_command_reply: tid=0 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 client.15888 shutdown 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 msgr2=0x560728882590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0be7e0 tx=0x7fac3c0be620 comp rx=0 tx=0).stop 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 shutdown_connections 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x560728894a20 0x560728896ea0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728882590 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728882050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac23fff640 0 client.15888 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 --2- 172.21.15.105:0/144567813 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728881b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 >> 172.21.15.105:0/144567813 conn(0x560728875cb0 msgr2=0x560728877a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 shutdown_connections 2024-04-22T06:38:14.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 1 -- 172.21.15.105:0/144567813 wait complete. 2024-04-22T06:38:14.352 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.345+0000 7fac4da77740 20 client.15888 trim_cache size 0 max 16384 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 Processor -- start 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 -- start start 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728884240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728890730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fac441ad940 con 0x560728884a80 2024-04-22T06:38:14.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fac441af480 con 0x56072887bea0 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4da77740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac441ae010 con 0x56072887b580 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32858/0 (socket says 172.21.15.105:32858) 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 -- 172.21.15.105:0/2406144652 learned_addr learned my addr 172.21.15.105:0/2406144652 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac3bfff640 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728890730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac48c1d640 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728884240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 -- 172.21.15.105:0/2406144652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 msgr2=0x560728884240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728884240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 -- 172.21.15.105:0/2406144652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 msgr2=0x560728890730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728890730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 -- 172.21.15.105:0/2406144652 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56072887bcd0 con 0x560728884a80 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.363+0000 7fac4941e640 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0bcd90 tx=0x7fac3c0be750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509af374176014cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac2a7fc640 1 -- 172.21.15.105:0/2406144652 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0be440 con 0x560728884a80 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac2a7fc640 1 -- 172.21.15.105:0/2406144652 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c035e50 con 0x560728884a80 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac2a7fc640 1 -- 172.21.15.105:0/2406144652 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0327b0 con 0x560728884a80 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 -- 172.21.15.105:0/2406144652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 msgr2=0x560728880c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0bcd90 tx=0x7fac3c0be750 comp rx=0 tx=0).stop 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 -- 172.21.15.105:0/2406144652 shutdown_connections 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x560728884a80 0x560728880c70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728890730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 --2- 172.21.15.105:0/2406144652 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56072887b580 0x560728884240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 -- 172.21.15.105:0/2406144652 >> 172.21.15.105:0/2406144652 conn(0x560728875cb0 msgr2=0x5607288770e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 -- 172.21.15.105:0/2406144652 shutdown_connections 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.364+0000 7fac4da77740 1 -- 172.21.15.105:0/2406144652 wait complete. 2024-04-22T06:38:14.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 -- start start 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 Processor -- start 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728888450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728888a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fac441aedb0 con 0x56072887b580 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fac441ae6e0 con 0x56072887bea0 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fac441ab9a0 con 0x560728884a80 2024-04-22T06:38:14.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac4da77740 10 client.0 _ll_register_callbacks cb 0x7fac3c0bad80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32862/0 (socket says 172.21.15.105:32862) 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 -- 172.21.15.105:0/3721758495 learned_addr learned my addr 172.21.15.105:0/3721758495 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728888450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 -- 172.21.15.105:0/3721758495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 msgr2=0x560728888a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728888a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 -- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 msgr2=0x560728888450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728888450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac48c1d640 1 -- 172.21.15.105:0/3721758495 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560728889230 con 0x56072887b580 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.365+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728888450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.372 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac48c1d640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x5607288976b0 tx=0x7fac40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b8d5481e9225ba server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40011e60 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fac40028740 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac3c035e50 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac40028940 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 10 client.15894 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607288fb120 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac3c035e50 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fac40039050 con 0x56072887b580 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 10 client.15894 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 20 client.15894 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.373 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5607288fb850 0x5607288fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 4 client.15894 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607287e7a70 con 0x5607288fb850 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.366+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5607288fb850 0x5607288fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac3bfff640 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5607288fb850 0x5607288fdc70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efb18cfa247a7644 server_cookie=34b04926706491f9 in_seq=0 out_seq=0 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac23fff640 10 client.15894 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac23fff640 1 -- 172.21.15.105:0/3721758495 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5607287e7a70 con 0x5607288fb850 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac23fff640 10 client.15894 handle_command_reply: tid=0 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 client.15894 shutdown 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 msgr2=0x56072888d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x5607288976b0 tx=0x7fac40004730 comp rx=0 tx=0).stop 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 shutdown_connections 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5607288fb850 0x5607288fdc70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x560728884a80 0x560728888a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56072887bea0 0x560728888450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 --2- 172.21.15.105:0/3721758495 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56072887b580 0x56072888d0e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 >> 172.21.15.105:0/3721758495 conn(0x560728875cb0 msgr2=0x560728877970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac23fff640 0 client.15894 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 shutdown_connections 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 1 -- 172.21.15.105:0/3721758495 wait complete. 2024-04-22T06:38:14.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.367+0000 7fac4da77740 20 client.15894 trim_cache size 0 max 16384 2024-04-22T06:38:14.377 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:14.378 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "no active scrubs running", 2024-04-22T06:38:14.378 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": {} 2024-04-22T06:38:14.378 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 msgr2=0x7fac2c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 0x7fac2c0a6390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fac4000bbe0 tx=0x7fac4000b390 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 msgr2=0x7fac4419d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005e30 tx=0x7fac3c034760 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 shutdown_connections 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fac2c0a3ed0 0x7fac2c0a6390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac4419cfd0 0x7fac4419d480 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fac44107240 0x7fac44198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 --2- 172.21.15.105:0/3042367865 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac44102170 0x7fac44198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 >> 172.21.15.105:0/3042367865 conn(0x7fac440793a0 msgr2=0x7fac440ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 shutdown_connections 2024-04-22T06:38:14.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.368+0000 7fac4ae81640 1 -- 172.21.15.105:0/3042367865 wait complete. 2024-04-22T06:38:14.380 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:14 smithi105 ceph-mon[27916]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 29 op/s 2024-04-22T06:38:14.381 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:38:14.381 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:38:14.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:14 smithi112 ceph-mon[31236]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 29 op/s 2024-04-22T06:38:14.481 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- 172.21.15.105:0/2317755536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734105ea0 msgr2=0x7f4734106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.481 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 --2- 172.21.15.105:0/2317755536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734105ea0 0x7f4734106280 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f472c009900 tx=0x7f472c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- 172.21.15.105:0/2317755536 shutdown_connections 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 --2- 172.21.15.105:0/2317755536 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734101b40 0x7f473410dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 --2- 172.21.15.105:0/2317755536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 0x7f4734101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 --2- 172.21.15.105:0/2317755536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734105ea0 0x7f4734106280 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- 172.21.15.105:0/2317755536 >> 172.21.15.105:0/2317755536 conn(0x7f4734077e30 msgr2=0x7f4734078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- 172.21.15.105:0/2317755536 shutdown_connections 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- 172.21.15.105:0/2317755536 wait complete. 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 Processor -- start 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.481+0000 7f4739123640 1 -- start start 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 0x7f473419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.482 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734105ea0 0x7f47341985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4734112260 con 0x7f4734101b40 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47341120e0 con 0x7f47341011a0 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f47341123e0 con 0x7f4734105ea0 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:32888/0 (socket says 172.21.15.105:32888) 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 -- 172.21.15.105:0/2179631485 learned_addr learned my addr 172.21.15.105:0/2179631485 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4732d9d640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 0x7f473419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473359e640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734105ea0 0x7f47341985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 -- 172.21.15.105:0/2179631485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734105ea0 msgr2=0x7f47341985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734105ea0 0x7f47341985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.483 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 -- 172.21.15.105:0/2179631485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 msgr2=0x7f473419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 0x7f473419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 -- 172.21.15.105:0/2179631485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4734198d60 con 0x7f4734101b40 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f473259c640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f471c00e9f0 tx=0x7f471c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c00cd80 con 0x7f4734101b40 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4734199050 con 0x7f4734101b40 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f471c0108e0 con 0x7f4734101b40 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47341a52a0 con 0x7f4734101b40 2024-04-22T06:38:14.484 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.482+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c018920 con 0x7f4734101b40 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.483+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4734102590 con 0x7f4734101b40 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.485+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f471c010430 con 0x7f4734101b40 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.485+0000 7f472bfff640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 0x7f47200a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.485+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f471c0d1000 con 0x7f4734101b40 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.485+0000 7f4732d9d640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 0x7f47200a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.485 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.486+0000 7f4732d9d640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 0x7f47200a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f472c006010 tx=0x7f472c02c6c0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.489+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f471c09d4d0 con 0x7f4734101b40 2024-04-22T06:38:14.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:14 smithi177 ceph-mon[31346]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 29 op/s 2024-04-22T06:38:14.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.705+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4734069230 con 0x7f4734101b40 2024-04-22T06:38:14.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.706+0000 7f472bfff640 1 -- 172.21.15.105:0/2179631485 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f471c09cdd0 con 0x7f4734101b40 2024-04-22T06:38:14.705 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:38:14.705 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:38:14.705 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 msgr2=0x7f47200a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 0x7f47200a6390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f472c006010 tx=0x7f472c02c6c0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 msgr2=0x7f473419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f471c00e9f0 tx=0x7f471c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 shutdown_connections 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f47200a3ed0 0x7f47200a6390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4734105ea0 0x7f47341985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4734101b40 0x7f473419e920 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 --2- 172.21.15.105:0/2179631485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47341011a0 0x7f473419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 >> 172.21.15.105:0/2179631485 conn(0x7f4734077e30 msgr2=0x7f47340feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 shutdown_connections 2024-04-22T06:38:14.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.708+0000 7f4739123640 1 -- 172.21.15.105:0/2179631485 wait complete. 2024-04-22T06:38:14.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T06:38:14.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- 172.21.15.105:0/494461928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c102190 msgr2=0x7f551c102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 --2- 172.21.15.105:0/494461928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c102190 0x7f551c102570 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5514009ca0 tx=0x7f551402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- 172.21.15.105:0/494461928 shutdown_connections 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 --2- 172.21.15.105:0/494461928 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c107260 0x7f551c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 --2- 172.21.15.105:0/494461928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c102b40 0x7f551c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 --2- 172.21.15.105:0/494461928 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c102190 0x7f551c102570 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- 172.21.15.105:0/494461928 >> 172.21.15.105:0/494461928 conn(0x7f551c0793a0 msgr2=0x7f551c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- 172.21.15.105:0/494461928 shutdown_connections 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- 172.21.15.105:0/494461928 wait complete. 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 Processor -- start 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.819+0000 7f552410e640 1 -- start start 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c102190 0x7f551c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c107260 0x7f551c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1121c0 con 0x7f551c19cfe0 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f551c112040 con 0x7f551c102190 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f551c112340 con 0x7f551c107260 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47426/0 (socket says 172.21.15.105:47426) 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 -- 172.21.15.105:0/3953824089 learned_addr learned my addr 172.21.15.105:0/3953824089 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f5521eaa640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c102190 0x7f551c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55216a9640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c107260 0x7f551c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 -- 172.21.15.105:0/3953824089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c107260 msgr2=0x7f551c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c107260 0x7f551c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 -- 172.21.15.105:0/3953824089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c102190 msgr2=0x7f551c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c102190 0x7f551c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 -- 172.21.15.105:0/3953824089 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c19dad0 con 0x7f551c19cfe0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f55226ab640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c970 tx=0x7f550c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f551affd640 1 -- 172.21.15.105:0/3953824089 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c007bb0 con 0x7f551c19cfe0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c19ddc0 con 0x7f551c19cfe0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f551affd640 1 -- 172.21.15.105:0/3953824089 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f550c007d50 con 0x7f551c19cfe0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.820+0000 7f551affd640 1 -- 172.21.15.105:0/3953824089 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c018770 con 0x7f551c19cfe0 2024-04-22T06:38:14.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.821+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c1a55b0 con 0x7f551c19cfe0 2024-04-22T06:38:14.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.822+0000 7f551affd640 1 -- 172.21.15.105:0/3953824089 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f550c00f990 con 0x7f551c19cfe0 2024-04-22T06:38:14.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 Processor -- start 2024-04-22T06:38:14.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 -- start start 2024-04-22T06:38:14.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f10e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559322e6a1f0 con 0x559322f054d0 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559322e074b0 con 0x559322f05e80 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5526ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559322e661b0 con 0x559322f0ea00 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f551affd640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 0x7f55040a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f55216a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f55216a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47442/0 (socket says 172.21.15.105:47442) 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f55216a9640 1 -- 172.21.15.105:0/4154967971 learned_addr learned my addr 172.21.15.105:0/4154967971 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f55226ab640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 0x7f55040a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.823+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f10e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f551affd640 1 -- 172.21.15.105:0/3953824089 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f550c0d0b00 con 0x7f551c19cfe0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 -- 172.21.15.105:0/4154967971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 msgr2=0x559322f10e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f10e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 -- 172.21.15.105:0/4154967971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 msgr2=0x559322f0e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 -- 172.21.15.105:0/4154967971 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559322df91b0 con 0x559322f054d0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55216a9640 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f55100134a0 tx=0x7f5510009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990350b46d441a2c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f55226ab640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 0x7f55040a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f550c0ce0e0 tx=0x7f550c08e040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f5503fff640 1 -- 172.21.15.105:0/4154967971 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510013270 con 0x559322f054d0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f5503fff640 1 -- 172.21.15.105:0/4154967971 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5510010940 con 0x559322f054d0 2024-04-22T06:38:14.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.824+0000 7f5503fff640 1 -- 172.21.15.105:0/4154967971 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551000f940 con 0x559322f054d0 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- 172.21.15.105:0/4154967971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 msgr2=0x559322f058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f55100134a0 tx=0x7f5510009760 comp rx=0 tx=0).stop 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- 172.21.15.105:0/4154967971 shutdown_connections 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f10e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4154967971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f058b0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- 172.21.15.105:0/4154967971 >> 172.21.15.105:0/4154967971 conn(0x559322effba0 msgr2=0x559322efff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- 172.21.15.105:0/4154967971 shutdown_connections 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- 172.21.15.105:0/4154967971 wait complete. 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 -- start start 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.825+0000 7f5526ddc740 1 Processor -- start 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559322f11910 con 0x559322f054d0 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559322e23500 con 0x559322f05e80 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559322f11a90 con 0x559322f0ea00 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5526ddc740 10 client.0 _ll_register_callbacks cb 0x559322df1b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47448/0 (socket says 172.21.15.105:47448) 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 -- 172.21.15.105:0/222089846 learned_addr learned my addr 172.21.15.105:0/222089846 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f5521eaa640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 -- 172.21.15.105:0/222089846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 msgr2=0x559322f0bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 -- 172.21.15.105:0/222089846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 msgr2=0x559322f0b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 -- 172.21.15.105:0/222089846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559322b7c3f0 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.826+0000 7f55216a9640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5510006010 tx=0x7f551000f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b463751d09b0779b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55100107a0 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f55100054d0 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5510010940 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5510013070 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 10 client.15912 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559322c467d0 con 0x559322f054d0 2024-04-22T06:38:14.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5510010940 con 0x559322f054d0 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f551001e070 con 0x559322f054d0 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 10 client.15912 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 10 client.15912 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 20 client.15912 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f1ea60 0x559322f20e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 4 client.15912 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559322e09930 con 0x559322f1ea60 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.827+0000 7f5521eaa640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f1ea60 0x559322f20e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f55017fa640 10 client.15912 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f5521eaa640 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f1ea60 0x559322f20e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c55821392494b81 server_cookie=cffd4087373d21db in_seq=0 out_seq=0 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f55017fa640 1 -- 172.21.15.105:0/222089846 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559322e09930 con 0x559322f1ea60 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f55017fa640 10 client.15912 handle_command_reply: tid=0 2024-04-22T06:38:14.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f5526ddc740 1 client.15912 shutdown 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 msgr2=0x559322f0e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.828+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5510006010 tx=0x7f551000f5c0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 shutdown_connections 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f1ea60 0x559322f20e50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f05e80 0x559322f0b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 --2- 172.21.15.105:0/222089846 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f054d0 0x559322f0e340 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f55017fa640 0 client.15912 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 >> 172.21.15.105:0/222089846 conn(0x559322effba0 msgr2=0x559322f018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 shutdown_connections 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 1 -- 172.21.15.105:0/222089846 wait complete. 2024-04-22T06:38:14.836 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.829+0000 7f5526ddc740 20 client.15912 trim_cache size 0 max 16384 2024-04-22T06:38:14.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 Processor -- start 2024-04-22T06:38:14.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 -- start start 2024-04-22T06:38:14.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f1a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f170c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1ad9c0 con 0x559322f05e80 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1af500 con 0x559322f054d0 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5526ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1ae090 con 0x559322f0ea00 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47464/0 (socket says 172.21.15.105:47464) 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 -- 172.21.15.105:0/4005161597 learned_addr learned my addr 172.21.15.105:0/4005161597 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 -- 172.21.15.105:0/4005161597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 msgr2=0x559322f170c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f170c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 -- 172.21.15.105:0/4005161597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 msgr2=0x559322f1a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f1a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 -- 172.21.15.105:0/4005161597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559322eff220 con 0x559322f05e80 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.842+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f551402c630 tx=0x7f5514005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd0decc5465de8f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f55037fe640 1 -- 172.21.15.105:0/4005161597 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551404f070 con 0x559322f05e80 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f55037fe640 1 -- 172.21.15.105:0/4005161597 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5514032970 con 0x559322f05e80 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f55037fe640 1 -- 172.21.15.105:0/4005161597 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514039040 con 0x559322f05e80 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 -- 172.21.15.105:0/4005161597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 msgr2=0x559322efe370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f551402c630 tx=0x7f5514005c50 comp rx=0 tx=0).stop 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 -- 172.21.15.105:0/4005161597 shutdown_connections 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f170c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322efe370 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4005161597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f1a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 -- 172.21.15.105:0/4005161597 >> 172.21.15.105:0/4005161597 conn(0x559322effba0 msgr2=0x559322f01730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 -- 172.21.15.105:0/4005161597 shutdown_connections 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.843+0000 7f5526ddc740 1 -- 172.21.15.105:0/4005161597 wait complete. 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 -- start start 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 Processor -- start 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f80700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1aee30 con 0x559322f05e80 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1ae760 con 0x559322f054d0 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f551c1aba00 con 0x559322f0ea00 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5526ddc740 10 client.0 _ll_register_callbacks cb 0x7f5510010f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47474/0 (socket says 172.21.15.105:47474) 2024-04-22T06:38:14.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 -- 172.21.15.105:0/4031030732 learned_addr learned my addr 172.21.15.105:0/4031030732 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f55216a9640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f80700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 -- 172.21.15.105:0/4031030732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 msgr2=0x559322f0d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 -- 172.21.15.105:0/4031030732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 msgr2=0x559322f80700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f80700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f5521eaa640 1 -- 172.21.15.105:0/4031030732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559322f0dd80 con 0x559322f05e80 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.844+0000 7f55216a9640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f80700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5521eaa640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5514037040 tx=0x7f5514009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ce9642cc9e3217 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551404f070 con 0x559322f05e80 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5514032970 con 0x559322f05e80 2024-04-22T06:38:14.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5514032d80 con 0x559322f05e80 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5514039040 con 0x559322f05e80 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 10 client.15918 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559322f88ff0 con 0x559322f05e80 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5514032970 con 0x559322f05e80 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f55140041b0 con 0x559322f05e80 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 10 client.15918 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 10 client.15918 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 20 client.15918 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f896c0 0x559322f8bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 4 client.15918 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 --> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559322e77c10 con 0x559322f896c0 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.845+0000 7f55216a9640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f896c0 0x559322f8bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f55017fa640 10 client.15918 ms_handle_connect on v2:172.21.15.105:6840/2147123680 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f55216a9640 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f896c0 0x559322f8bae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dde4e694c62c5a2b server_cookie=cc0105207dee7ea4 in_seq=0 out_seq=0 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f55017fa640 1 -- 172.21.15.105:0/4031030732 <== mds.4 v2:172.21.15.105:6840/2147123680 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559322e77c10 con 0x559322f896c0 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f55017fa640 10 client.15918 handle_command_reply: tid=0 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 client.15918 shutdown 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 msgr2=0x559322f80c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5514037040 tx=0x7f5514009940 comp rx=0 tx=0).stop 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 shutdown_connections 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:6840/2147123680,v1:172.21.15.105:6841/2147123680] conn(0x559322f896c0 0x559322f8bae0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559322f0ea00 0x559322f0d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559322f05e80 0x559322f80c40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 --2- 172.21.15.105:0/4031030732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559322f054d0 0x559322f80700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f55017fa640 0 client.15918 ms_handle_reset on v2:172.21.15.105:6840/2147123680 2024-04-22T06:38:14.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 >> 172.21.15.105:0/4031030732 conn(0x559322effba0 msgr2=0x559322f06260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 shutdown_connections 2024-04-22T06:38:14.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 1 -- 172.21.15.105:0/4031030732 wait complete. 2024-04-22T06:38:14.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.846+0000 7f5526ddc740 20 client.15918 trim_cache size 0 max 16384 2024-04-22T06:38:14.853 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 msgr2=0x7f55040a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 0x7f55040a6390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f550c0ce0e0 tx=0x7f550c08e040 comp rx=0 tx=0).stop 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 msgr2=0x7f551c19d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c970 tx=0x7f550c00ce30 comp rx=0 tx=0).stop 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 shutdown_connections 2024-04-22T06:38:14.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55040a3ed0 0x7f55040a6390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f551c19cfe0 0x7f551c19d490 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f551c107260 0x7f551c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 --2- 172.21.15.105:0/3953824089 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f551c102190 0x7f551c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 >> 172.21.15.105:0/3953824089 conn(0x7f551c0793a0 msgr2=0x7f551c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 shutdown_connections 2024-04-22T06:38:14.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.847+0000 7f552410e640 1 -- 172.21.15.105:0/3953824089 wait complete. 2024-04-22T06:38:14.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T06:38:14.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.957+0000 7f8d115b2640 1 -- 172.21.15.105:0/812343946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c101090 msgr2=0x7f8d0c101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.957+0000 7f8d115b2640 1 --2- 172.21.15.105:0/812343946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c101090 0x7f8d0c101470 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009ca0 tx=0x7f8cfc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.957+0000 7f8d115b2640 1 -- 172.21.15.105:0/812343946 shutdown_connections 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.957+0000 7f8d115b2640 1 --2- 172.21.15.105:0/812343946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c106160 0x7f8d0c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.957+0000 7f8d115b2640 1 --2- 172.21.15.105:0/812343946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d0c101a40 0x7f8d0c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 --2- 172.21.15.105:0/812343946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c101090 0x7f8d0c101470 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- 172.21.15.105:0/812343946 >> 172.21.15.105:0/812343946 conn(0x7f8d0c077f50 msgr2=0x7f8d0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- 172.21.15.105:0/812343946 shutdown_connections 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- 172.21.15.105:0/812343946 wait complete. 2024-04-22T06:38:14.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 Processor -- start 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- start start 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 0x7f8d0c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d0c19cfc0 0x7f8d0c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1121e0 con 0x7f8d0c106160 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c112060 con 0x7f8d0c19cfc0 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.958+0000 7f8d115b2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c112360 con 0x7f8d0c101090 2024-04-22T06:38:14.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47496/0 (socket says 172.21.15.105:47496) 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/3686015758 learned_addr learned my addr 172.21.15.105:0/3686015758 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0affd640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 0x7f8d0c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/3686015758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 msgr2=0x7f8d0c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 0x7f8d0c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/3686015758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d0c19cfc0 msgr2=0x7f8d0c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d0c19cfc0 0x7f8d0c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/3686015758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c19dab0 con 0x7f8d0c106160 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0affd640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 0x7f8d0c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c970 tx=0x7f8d0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8cf3fff640 1 -- 172.21.15.105:0/3686015758 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00014070 con 0x7f8d0c106160 2024-04-22T06:38:14.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8cf3fff640 1 -- 172.21.15.105:0/3686015758 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d00004440 con 0x7f8d0c106160 2024-04-22T06:38:14.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c19dda0 con 0x7f8d0c106160 2024-04-22T06:38:14.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8cf3fff640 1 -- 172.21.15.105:0/3686015758 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00003e50 con 0x7f8d0c106160 2024-04-22T06:38:14.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.959+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c1a5590 con 0x7f8d0c106160 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.961+0000 7f8cf3fff640 1 -- 172.21.15.105:0/3686015758 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f8d00017070 con 0x7f8d0c106160 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 Processor -- start 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 -- start start 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a04d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a056430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.969 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f839fb21f0 con 0x55f83a056a00 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f839f4f4b0 con 0x55f83a04d4d0 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d141a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f839fae1b0 con 0x55f83a04de80 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8cf3fff640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 0x7f8cf40a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 0x7f8cf40a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47500/0 (socket says 172.21.15.105:47500) 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/1917786340 learned_addr learned my addr 172.21.15.105:0/1917786340 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a04d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0affd640 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a056430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8cf3fff640 1 -- 172.21.15.105:0/3686015758 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f8d000d06b0 con 0x7f8d0c106160 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/1917786340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 msgr2=0x55f83a056430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a056430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.970 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/1917786340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 msgr2=0x55f83a04d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a04d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.962+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/1917786340 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f839f411b0 con 0x55f83a056a00 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 0x7f8cf40a6530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d04013650 tx=0x7f8d0400e7f0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cdbd0 tx=0x7f8d000c7590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed62f66442df812 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8cf17fa640 1 -- 172.21.15.105:0/1917786340 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000de20 con 0x55f83a056a00 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8cf17fa640 1 -- 172.21.15.105:0/1917786340 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0009c480 con 0x55f83a056a00 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8cf17fa640 1 -- 172.21.15.105:0/1917786340 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d000981d0 con 0x55f83a056a00 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 -- 172.21.15.105:0/1917786340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 msgr2=0x55f83a058e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8d000cdbd0 tx=0x7f8d000c7590 comp rx=0 tx=0).stop 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 -- 172.21.15.105:0/1917786340 shutdown_connections 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a058e50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a056430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 --2- 172.21.15.105:0/1917786340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a04d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.971 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 -- 172.21.15.105:0/1917786340 >> 172.21.15.105:0/1917786340 conn(0x55f83a047ba0 msgr2=0x55f83a047f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.963+0000 7f8d141a5740 1 -- 172.21.15.105:0/1917786340 shutdown_connections 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 -- 172.21.15.105:0/1917786340 wait complete. 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 -- start start 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 Processor -- start 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 0x55f83a056350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a056a00 0x55f83a053cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f83a059910 con 0x55f83a04de80 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f839f6b500 con 0x55f83a056a00 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f83a059a90 con 0x55f83a04d4d0 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d141a5740 10 client.0 _ll_register_callbacks cb 0x55f839f39b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47514/0 (socket says 172.21.15.105:47514) 2024-04-22T06:38:14.972 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/2153611936 learned_addr learned my addr 172.21.15.105:0/2153611936 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a056a00 0x55f83a053cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0affd640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 0x55f83a056350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 msgr2=0x55f83a056350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 0x55f83a056350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/2153611936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a056a00 msgr2=0x55f83a053cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a056a00 0x55f83a053cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/2153611936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f839cc43f0 con 0x55f83a04de80 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.964+0000 7f8d0affd640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 0x55f83a056350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c410 tx=0x7f8d00007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fdcecb703095a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d000a0830 con 0x55f83a04de80 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f8d000981d0 con 0x55f83a04de80 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d0009c480 con 0x55f83a04de80 2024-04-22T06:38:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d000a50f0 con 0x55f83a04de80 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 10 client.15930 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f839d8e7d0 con 0x55f83a04de80 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d0009c480 con 0x55f83a04de80 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d000cdf40 con 0x55f83a04de80 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 10 client.15930 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 10 client.15930 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 20 client.15930 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a066a80 0x55f83a068e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 4 client.15930 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.965+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f839f51930 con 0x55f83a066a80 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.966+0000 7f8d0affd640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a066a80 0x55f83a068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.966+0000 7f8d0affd640 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a066a80 0x55f83a068e70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa00e7c71ae09018 server_cookie=afa08e3aed340f65 in_seq=0 out_seq=0 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.966+0000 7f8ceeff5640 10 client.15930 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:38:14.974 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8ceeff5640 1 -- 172.21.15.105:0/2153611936 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f839f51930 con 0x55f83a066a80 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8ceeff5640 10 client.15930 handle_command_reply: tid=0 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 client.15930 shutdown 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 msgr2=0x55f83a0536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c410 tx=0x7f8d00007730 comp rx=0 tx=0).stop 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 shutdown_connections 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a066a80 0x55f83a068e70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a056a00 0x55f83a053cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a0536e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8ceeff5640 0 client.15930 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 --2- 172.21.15.105:0/2153611936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04d4d0 0x55f83a056350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 >> 172.21.15.105:0/2153611936 conn(0x55f83a047ba0 msgr2=0x55f83a0498f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 shutdown_connections 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 1 -- 172.21.15.105:0/2153611936 wait complete. 2024-04-22T06:38:14.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.967+0000 7f8d141a5740 20 client.15930 trim_cache size 0 max 16384 2024-04-22T06:38:14.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 Processor -- start 2024-04-22T06:38:14.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 -- start start 2024-04-22T06:38:14.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a055e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a056a00 0x55f83a06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1adab0 con 0x55f83a04de80 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1af5f0 con 0x55f83a04d4d0 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d141a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ae180 con 0x55f83a056a00 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47518/0 (socket says 172.21.15.105:47518) 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.980+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/586560758 learned_addr learned my addr 172.21.15.105:0/586560758 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0affd640 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/586560758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a056a00 msgr2=0x55f83a06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a056a00 0x55f83a06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/586560758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 msgr2=0x55f83a055e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a055e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 -- 172.21.15.105:0/586560758 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83a051c30 con 0x55f83a04de80 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0affd640 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a055e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c940 tx=0x7f8d00000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52cf586c0a582ff server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8cf0ff9640 1 -- 172.21.15.105:0/586560758 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000de20 con 0x55f83a04de80 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8cf0ff9640 1 -- 172.21.15.105:0/586560758 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d000a0830 con 0x55f83a04de80 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8cf0ff9640 1 -- 172.21.15.105:0/586560758 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0009b630 con 0x55f83a04de80 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d141a5740 1 -- 172.21.15.105:0/586560758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 msgr2=0x55f83a056350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.981+0000 7f8d141a5740 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c940 tx=0x7f8d00000bc0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- 172.21.15.105:0/586560758 shutdown_connections 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a056a00 0x55f83a06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a04de80 0x55f83a056350 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- 172.21.15.105:0/586560758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a055e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- 172.21.15.105:0/586560758 >> 172.21.15.105:0/586560758 conn(0x55f83a047ba0 msgr2=0x55f83a049880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- 172.21.15.105:0/586560758 shutdown_connections 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- 172.21.15.105:0/586560758 wait complete. 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- start start 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 Processor -- start 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a06a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1aef20 con 0x55f83a056a00 2024-04-22T06:38:14.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ae850 con 0x55f83a04d4d0 2024-04-22T06:38:14.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1abab0 con 0x55f83a04de80 2024-04-22T06:38:14.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d141a5740 10 client.0 _ll_register_callbacks cb 0x7f8d000d0930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:14.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47524/0 (socket says 172.21.15.105:47524) 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 -- 172.21.15.105:0/3726109654 learned_addr learned my addr 172.21.15.105:0/3726109654 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0a7fc640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0affd640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a06a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 -- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 msgr2=0x55f83a06a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a06a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 -- 172.21.15.105:0/3726109654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 msgr2=0x55f83a06a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a06a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.982+0000 7f8d0b7fe640 1 -- 172.21.15.105:0/3726109654 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f83a0c6220 con 0x55f83a056a00 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d0b7fe640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8d04013ee0 tx=0x7f8d040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd5c1abd7e5b102 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:14.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0401f7b0 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f8d04009070 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d000a0830 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8d04004020 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 10 client.15936 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f83a0cd530 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d000a0830 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8d0400f040 con 0x55f83a056a00 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 10 client.15936 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 10 client.15936 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 20 client.15936 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a0cdce0 0x55f83a0d0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 4 client.15936 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.983+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 --> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f839fbfc10 con 0x55f83a0cdce0 2024-04-22T06:38:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d0affd640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a0cdce0 0x55f83a0d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d0affd640 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a0cdce0 0x55f83a0d0100 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9127fc896bcd46a server_cookie=926856c736938de4 in_seq=0 out_seq=0 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8ceeff5640 10 client.15936 ms_handle_connect on v2:172.21.15.177:6836/4114728122 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8ceeff5640 1 -- 172.21.15.105:0/3726109654 <== mds.3 v2:172.21.15.177:6836/4114728122 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f839fbfc10 con 0x55f83a0cdce0 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8ceeff5640 10 client.15936 handle_command_reply: tid=0 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 client.15936 shutdown 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 msgr2=0x55f83a0c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8d04013ee0 tx=0x7f8d040046c0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 shutdown_connections 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:6836/4114728122,v1:172.21.15.177:6837/4114728122] conn(0x55f83a0cdce0 0x55f83a0d0100 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f83a056a00 0x55f83a0c5c30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f83a04de80 0x55f83a06a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8ceeff5640 0 client.15936 ms_handle_reset on v2:172.21.15.177:6836/4114728122 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 --2- 172.21.15.105:0/3726109654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f83a04d4d0 0x55f83a06a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 >> 172.21.15.105:0/3726109654 conn(0x55f83a047ba0 msgr2=0x55f83a048fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 shutdown_connections 2024-04-22T06:38:14.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 1 -- 172.21.15.105:0/3726109654 wait complete. 2024-04-22T06:38:14.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.984+0000 7f8d141a5740 20 client.15936 trim_cache size 0 max 16384 2024-04-22T06:38:14.993 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:14.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 msgr2=0x7f8cf40a6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 0x7f8cf40a6530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8d04013650 tx=0x7f8d0400e7f0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 msgr2=0x7f8d0c198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c970 tx=0x7f8d0000ce30 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 shutdown_connections 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f8cf40a4070 0x7f8cf40a6530 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d0c19cfc0 0x7f8d0c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d0c106160 0x7f8d0c198c40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 --2- 172.21.15.105:0/3686015758 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d0c101090 0x7f8d0c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 >> 172.21.15.105:0/3686015758 conn(0x7f8d0c077f50 msgr2=0x7f8d0c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 shutdown_connections 2024-04-22T06:38:14.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:14.985+0000 7f8d115b2640 1 -- 172.21.15.105:0/3686015758 wait complete. 2024-04-22T06:38:14.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:38:15.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- 172.21.15.105:0/1050309640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5990102b50 msgr2=0x7f5990106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 --2- 172.21.15.105:0/1050309640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5990102b50 0x7f5990106b90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5980009900 tx=0x7f598002bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- 172.21.15.105:0/1050309640 shutdown_connections 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 --2- 172.21.15.105:0/1050309640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5990107270 0x7f599010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 --2- 172.21.15.105:0/1050309640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5990102b50 0x7f5990106b90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 --2- 172.21.15.105:0/1050309640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f59901021a0 0x7f5990102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- 172.21.15.105:0/1050309640 >> 172.21.15.105:0/1050309640 conn(0x7f5990079350 msgr2=0x7f5990079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- 172.21.15.105:0/1050309640 shutdown_connections 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- 172.21.15.105:0/1050309640 wait complete. 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 Processor -- start 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.098+0000 7f59976a8640 1 -- start start 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 0x7f5990198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f599019cf90 0x7f599019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59901121b0 con 0x7f59901021a0 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5990112030 con 0x7f5990107270 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5990112330 con 0x7f599019cf90 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47542/0 (socket says 172.21.15.105:47542) 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 -- 172.21.15.105:0/865688081 learned_addr learned my addr 172.21.15.105:0/865688081 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995c45640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f599019cf90 0x7f599019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5994c43640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 0x7f5990198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 -- 172.21.15.105:0/865688081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f599019cf90 msgr2=0x7f599019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f599019cf90 0x7f599019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 -- 172.21.15.105:0/865688081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 msgr2=0x7f5990198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 0x7f5990198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 -- 172.21.15.105:0/865688081 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599019da80 con 0x7f59901021a0 2024-04-22T06:38:15.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5994c43640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 0x7f5990198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f5995444640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f598800cce0 tx=0x7f598800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59867fc640 1 -- 172.21.15.105:0/865688081 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598800f650 con 0x7f59901021a0 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59867fc640 1 -- 172.21.15.105:0/865688081 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f598800fca0 con 0x7f59901021a0 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599019dd70 con 0x7f59901021a0 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59867fc640 1 -- 172.21.15.105:0/865688081 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988018b40 con 0x7f59901021a0 2024-04-22T06:38:15.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.099+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59901a5530 con 0x7f59901021a0 2024-04-22T06:38:15.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.101+0000 7f59867fc640 1 -- 172.21.15.105:0/865688081 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f598800f7f0 con 0x7f59901021a0 2024-04-22T06:38:15.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 Processor -- start 2024-04-22T06:38:15.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 -- start start 2024-04-22T06:38:15.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f59867fc640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 0x7f597c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.107 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e883de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625e87971f0 con 0x5625e8832e80 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5625e87344b0 con 0x5625e883ba00 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f599a379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5625e87931b0 con 0x5625e88324d0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f5994c43640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f5995c45640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 0x7f597c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.102+0000 7f5994c43640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47556/0 (socket says 172.21.15.105:47556) 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 -- 172.21.15.105:0/4059923304 learned_addr learned my addr 172.21.15.105:0/4059923304 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f59867fc640 1 -- 172.21.15.105:0/865688081 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f59880d0f00 con 0x7f59901021a0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5995444640 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5995c45640 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e883de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 -- 172.21.15.105:0/4059923304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 msgr2=0x5625e88328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88328b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 -- 172.21.15.105:0/4059923304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 msgr2=0x5625e883de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e883de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 -- 172.21.15.105:0/4059923304 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e87261b0 con 0x5625e8832e80 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5995c45640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 0x7f597c0a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f598c009ca0 tx=0x7f598c009830 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f5994c43640 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5980000c00 tx=0x7f598002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b31456e52c65d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f597b7fe640 1 -- 172.21.15.105:0/4059923304 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980044020 con 0x5625e8832e80 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f597b7fe640 1 -- 172.21.15.105:0/4059923304 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f598002cc90 con 0x5625e8832e80 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.103+0000 7f597b7fe640 1 -- 172.21.15.105:0/4059923304 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800046c0 con 0x5625e8832e80 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- 172.21.15.105:0/4059923304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 msgr2=0x5625e883b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5980000c00 tx=0x7f598002c650 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- 172.21.15.105:0/4059923304 shutdown_connections 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e883de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e883b430 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 --2- 172.21.15.105:0/4059923304 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88328b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- 172.21.15.105:0/4059923304 >> 172.21.15.105:0/4059923304 conn(0x5625e882cba0 msgr2=0x5625e882cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- 172.21.15.105:0/4059923304 shutdown_connections 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- 172.21.15.105:0/4059923304 wait complete. 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 -- start start 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.104+0000 7f599a379740 1 Processor -- start 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e883b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e8838b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5625e883e910 con 0x5625e8832e80 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5625e8750500 con 0x5625e883ba00 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5625e883ea90 con 0x5625e88324d0 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 10 client.0 _ll_register_callbacks cb 0x5625e871eb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47562/0 (socket says 172.21.15.105:47562) 2024-04-22T06:38:15.110 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 -- 172.21.15.105:0/346177060 learned_addr learned my addr 172.21.15.105:0/346177060 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5995444640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e883b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5995c45640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e8838b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 -- 172.21.15.105:0/346177060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 msgr2=0x5625e883b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e883b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 -- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 msgr2=0x5625e8838b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e8838b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 -- 172.21.15.105:0/346177060 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e84a93f0 con 0x5625e8832e80 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5994c43640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5980045000 tx=0x7f598002c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533224c535510d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5978ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598004f070 con 0x5625e8832e80 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f59800037f0 con 0x5625e8832e80 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.105+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598002cc90 con 0x5625e8832e80 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f598005b050 con 0x5625e8832e80 2024-04-22T06:38:15.111 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 10 client.15948 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625e85737d0 con 0x5625e8832e80 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f598002cc90 con 0x5625e8832e80 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5980040a80 con 0x5625e8832e80 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 10 client.15948 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 10 client.15948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 20 client.15948 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e884b9a0 0x5625e884dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 4 client.15948 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625e8736930 con 0x5625e884b9a0 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.106+0000 7f5995444640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e884b9a0 0x5625e884dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f5978ff9640 10 client.15948 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f5995444640 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e884b9a0 0x5625e884dd90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=31c3e22d9f5d5fff server_cookie=3fc8e593bd8ddd82 in_seq=0 out_seq=0 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f5978ff9640 1 -- 172.21.15.105:0/346177060 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625e8736930 con 0x5625e884b9a0 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f5978ff9640 10 client.15948 handle_command_reply: tid=0 2024-04-22T06:38:15.112 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f599a379740 1 client.15948 shutdown 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 msgr2=0x5625e88385e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.107+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f5980045000 tx=0x7f598002c740 comp rx=0 tx=0).stop 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 shutdown_connections 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e883ba00 0x5625e8838b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e8832e80 0x5625e88385e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e884b9a0 0x5625e884dd90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 --2- 172.21.15.105:0/346177060 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e883b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f5978ff9640 0 client.15948 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 >> 172.21.15.105:0/346177060 conn(0x5625e882cba0 msgr2=0x5625e882e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 shutdown_connections 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 1 -- 172.21.15.105:0/346177060 wait complete. 2024-04-22T06:38:15.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.108+0000 7f599a379740 20 client.15948 trim_cache size 0 max 16384 2024-04-22T06:38:15.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 Processor -- start 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 -- start start 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e88324d0 0x5625e88511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e8832e80 0x5625e883ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59901ad9d0 con 0x5625e883ba00 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59901af510 con 0x5625e88324d0 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f599a379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59901ae0a0 con 0x5625e8832e80 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47568/0 (socket says 172.21.15.105:47568) 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 -- 172.21.15.105:0/959106380 learned_addr learned my addr 172.21.15.105:0/959106380 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995444640 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e88324d0 0x5625e88511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 -- 172.21.15.105:0/959106380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e8832e80 msgr2=0x5625e883ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e8832e80 0x5625e883ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 -- 172.21.15.105:0/959106380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e88324d0 msgr2=0x5625e88511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e88324d0 0x5625e88511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 -- 172.21.15.105:0/959106380 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e883e2b0 con 0x5625e883ba00 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.121+0000 7f5995c45640 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f598c005890 tx=0x7f598c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2d84752ff338ae server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f597affd640 1 -- 172.21.15.105:0/959106380 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c00fda0 con 0x5625e883ba00 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f597affd640 1 -- 172.21.15.105:0/959106380 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f598c013170 con 0x5625e883ba00 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f597affd640 1 -- 172.21.15.105:0/959106380 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c010430 con 0x5625e883ba00 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- 172.21.15.105:0/959106380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 msgr2=0x5625e883b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f598c005890 tx=0x7f598c01f000 comp rx=0 tx=0).stop 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- 172.21.15.105:0/959106380 shutdown_connections 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e883b250 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e8832e80 0x5625e883ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 --2- 172.21.15.105:0/959106380 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e88324d0 0x5625e88511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- 172.21.15.105:0/959106380 >> 172.21.15.105:0/959106380 conn(0x5625e882cba0 msgr2=0x5625e88475a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- 172.21.15.105:0/959106380 shutdown_connections 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- 172.21.15.105:0/959106380 wait complete. 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.122+0000 7f599a379740 1 -- start start 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 Processor -- start 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e8832e80 0x5625e88aa070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f59901aee40 con 0x5625e883ba00 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59901ae770 con 0x5625e8832e80 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f59901ab9b0 con 0x5625e88324d0 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f599a379740 10 client.0 _ll_register_callbacks cb 0x7f5980004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47576/0 (socket says 172.21.15.105:47576) 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 -- 172.21.15.105:0/2029655821 learned_addr learned my addr 172.21.15.105:0/2029655821 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.127 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5994c43640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e8832e80 0x5625e88aa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995444640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 -- 172.21.15.105:0/2029655821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 msgr2=0x5625e88a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 -- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e8832e80 msgr2=0x5625e88aa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e8832e80 0x5625e88aa070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.123+0000 7f5995c45640 1 -- 172.21.15.105:0/2029655821 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e88aa930 con 0x5625e883ba00 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5995c45640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f598c014010 tx=0x7f598c003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946bfea82a89f859 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598c02c070 con 0x5625e883ba00 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f598c004050 con 0x5625e883ba00 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598c013170 con 0x5625e883ba00 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f598c038050 con 0x5625e883ba00 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 10 client.15954 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.128 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5625e88b61f0 con 0x5625e883ba00 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f598c013170 con 0x5625e883ba00 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f598c027040 con 0x5625e883ba00 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 10 client.15954 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 10 client.15954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 20 client.15954 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e88b6890 0x5625e88b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 4 client.15954 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 --> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625e87a4c10 con 0x5625e88b6890 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.124+0000 7f5995444640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e88b6890 0x5625e88b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f5978ff9640 10 client.15954 ms_handle_connect on v2:172.21.15.112:6836/3970644798 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f5995444640 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e88b6890 0x5625e88b8cb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7af34e900bbf414 server_cookie=793fead3c1a621e6 in_seq=0 out_seq=0 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f5978ff9640 1 -- 172.21.15.105:0/2029655821 <== mds.2 v2:172.21.15.112:6836/3970644798 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625e87a4c10 con 0x5625e88b6890 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f5978ff9640 10 client.15954 handle_command_reply: tid=0 2024-04-22T06:38:15.129 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 client.15954 shutdown 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 msgr2=0x5625e88b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f598c014010 tx=0x7f598c003ad0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 shutdown_connections 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5625e883ba00 0x5625e88b4bb0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5625e8832e80 0x5625e88aa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.112:6836/3970644798,v1:172.21.15.112:6837/3970644798] conn(0x5625e88b6890 0x5625e88b8cb0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f5978ff9640 0 client.15954 ms_handle_reset on v2:172.21.15.112:6836/3970644798 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 --2- 172.21.15.105:0/2029655821 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5625e88324d0 0x5625e88a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 >> 172.21.15.105:0/2029655821 conn(0x5625e882cba0 msgr2=0x5625e882b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 shutdown_connections 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 1 -- 172.21.15.105:0/2029655821 wait complete. 2024-04-22T06:38:15.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.125+0000 7f599a379740 20 client.15954 trim_cache size 0 max 16384 2024-04-22T06:38:15.131 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:15.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 msgr2=0x7f597c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 0x7f597c0a6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f598c009ca0 tx=0x7f598c009830 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 msgr2=0x7f59901986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f598800cce0 tx=0x7f598800a620 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 shutdown_connections 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f597c0a3ed0 0x7f597c0a6390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f599019cf90 0x7f599019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5990107270 0x7f5990198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 --2- 172.21.15.105:0/865688081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f59901021a0 0x7f59901986d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 >> 172.21.15.105:0/865688081 conn(0x7f5990079350 msgr2=0x7f59901000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 shutdown_connections 2024-04-22T06:38:15.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.126+0000 7f59976a8640 1 -- 172.21.15.105:0/865688081 wait complete. 2024-04-22T06:38:15.135 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/132431236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 msgr2=0x7f0ccc107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/132431236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc107410 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009930 tx=0x7f0cbc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/132431236 shutdown_connections 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/132431236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 0x7f0ccc10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/132431236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/132431236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc107410 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/132431236 >> 172.21.15.105:0/132431236 conn(0x7f0ccc076c50 msgr2=0x7f0ccc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/132431236 shutdown_connections 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.237+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/132431236 wait complete. 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 Processor -- start 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 -- start start 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 0x7f0ccc198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc112260 con 0x7f0ccc107030 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1120e0 con 0x7f0ccc101b40 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cd0ef2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1123e0 con 0x7f0ccc1011a0 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47602/0 (socket says 172.21.15.105:47602) 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3619078040 learned_addr learned my addr 172.21.15.105:0/3619078040 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 0x7f0ccc198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3619078040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 msgr2=0x7f0ccc1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3619078040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 msgr2=0x7f0ccc198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 0x7f0ccc198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3619078040 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ccc19d6d0 con 0x7f0ccc107030 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.238+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009fd0 tx=0x7f0cc400def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0cbb7fe640 1 -- 172.21.15.105:0/3619078040 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4017070 con 0x7f0ccc107030 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0cbb7fe640 1 -- 172.21.15.105:0/3619078040 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc4003e20 con 0x7f0ccc107030 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ccc19d9c0 con 0x7f0ccc107030 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0cbb7fe640 1 -- 172.21.15.105:0/3619078040 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4010460 con 0x7f0ccc107030 2024-04-22T06:38:15.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.239+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ccc1a5250 con 0x7f0ccc107030 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.241+0000 7f0cbb7fe640 1 -- 172.21.15.105:0/3619078040 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f0cc4004020 con 0x7f0ccc107030 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.241+0000 7f0cd3bc1740 1 Processor -- start 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.241+0000 7f0cd3bc1740 1 -- start start 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cbb7fe640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 0x7f0cb40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da6c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55566d9d11f0 con 0x55566da75e60 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55566d96e4b0 con 0x55566da6cd60 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55566d9cd1b0 con 0x55566da6c3b0 2024-04-22T06:38:15.249 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 0x7f0cb40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cbb7fe640 1 -- 172.21.15.105:0/3619078040 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f0cc40d07b0 con 0x7f0ccc107030 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47618/0 (socket says 172.21.15.105:47618) 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cca59c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da6c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 -- 172.21.15.105:0/1289922714 learned_addr learned my addr 172.21.15.105:0/1289922714 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 -- 172.21.15.105:0/1289922714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 msgr2=0x55566da6c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da6c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 -- 172.21.15.105:0/1289922714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 msgr2=0x55566da75890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 -- 172.21.15.105:0/1289922714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566d9601b0 con 0x55566da75e60 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 0x7f0cb40a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00058c0 tx=0x7f0cc00138f0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4003fd0 tx=0x7f0cc4098220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd3a4368035bd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.242+0000 7f0cb8ff9640 1 -- 172.21.15.105:0/1289922714 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4020070 con 0x55566da75e60 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cb8ff9640 1 -- 172.21.15.105:0/1289922714 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc4070430 con 0x55566da75e60 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cb8ff9640 1 -- 172.21.15.105:0/1289922714 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc409bc30 con 0x55566da75e60 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/1289922714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 msgr2=0x55566da762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4003fd0 tx=0x7f0cc4098220 comp rx=0 tx=0).stop 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/1289922714 shutdown_connections 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da762c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/1289922714 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da6c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/1289922714 >> 172.21.15.105:0/1289922714 conn(0x55566da66ba0 msgr2=0x55566da66f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/1289922714 shutdown_connections 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.243+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/1289922714 wait complete. 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 -- start start 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 Processor -- start 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da7c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da79600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55566da7cce0 con 0x55566da75e60 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55566d98a500 con 0x55566da6cd60 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55566da7ce60 con 0x55566da6c3b0 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cd3bc1740 10 client.0 _ll_register_callbacks cb 0x55566d958b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47634/0 (socket says 172.21.15.105:47634) 2024-04-22T06:38:15.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 -- 172.21.15.105:0/874994479 learned_addr learned my addr 172.21.15.105:0/874994479 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0cca59c640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da7c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 -- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 msgr2=0x55566da7c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da7c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 -- 172.21.15.105:0/874994479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 msgr2=0x55566da79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da79600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 -- 172.21.15.105:0/874994479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55566d6e33f0 con 0x55566da75e60 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.244+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400b6f0 tx=0x7f0cc4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b71213212b236f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4020070 con 0x55566da75e60 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f0cc409bbe0 con 0x55566da75e60 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc4070430 con 0x55566da75e60 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cc40a0930 con 0x55566da75e60 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 10 client.15966 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55566d7ad7d0 con 0x55566da75e60 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc4070430 con 0x55566da75e60 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cc4085ac0 con 0x55566da75e60 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 10 client.15966 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 10 client.15966 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 20 client.15966 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da89e20 0x55566da8c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 4 client.15966 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.245+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55566d970930 con 0x55566da89e20 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.246+0000 7f0cca59c640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da89e20 0x55566da8c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.246+0000 7f0cca59c640 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da89e20 0x55566da8c210 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb73e526ddc6f5ed server_cookie=3e33b58ff0dbeeba in_seq=0 out_seq=0 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.246+0000 7f0cb1ffb640 10 client.15966 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/874994479 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55566d970930 con 0x55566da89e20 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cb1ffb640 10 client.15966 handle_command_reply: tid=0 2024-04-22T06:38:15.254 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 client.15966 shutdown 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 msgr2=0x55566da79b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400b6f0 tx=0x7f0cc4003ba0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 shutdown_connections 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da89e20 0x55566da8c210 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da79b40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da79600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/874994479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6c3b0 0x55566da7c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 >> 172.21.15.105:0/874994479 conn(0x55566da66ba0 msgr2=0x55566da688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cb1ffb640 0 client.15966 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 shutdown_connections 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/874994479 wait complete. 2024-04-22T06:38:15.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.247+0000 7f0cd3bc1740 20 client.15966 trim_cache size 0 max 16384 2024-04-22T06:38:15.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 Processor -- start 2024-04-22T06:38:15.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 -- start start 2024-04-22T06:38:15.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da75e60 0x55566da785e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1ab7c0 con 0x55566da6c3b0 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1ad300 con 0x55566da6cd60 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1abe90 con 0x55566da75e60 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47648/0 (socket says 172.21.15.105:47648) 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 -- 172.21.15.105:0/3104496044 learned_addr learned my addr 172.21.15.105:0/3104496044 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 -- 172.21.15.105:0/3104496044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da75e60 msgr2=0x55566da785e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da75e60 0x55566da785e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 -- 172.21.15.105:0/3104496044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 msgr2=0x55566da75620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cca59c640 1 -- 172.21.15.105:0/3104496044 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566da66380 con 0x55566da6c3b0 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.260+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc005940 tx=0x7f0cbc0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544be5c33484d214 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cb3fff640 1 -- 172.21.15.105:0/3104496044 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc04d020 con 0x55566da6c3b0 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cb3fff640 1 -- 172.21.15.105:0/3104496044 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cbc02cc90 con 0x55566da6c3b0 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cb3fff640 1 -- 172.21.15.105:0/3104496044 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0497c0 con 0x55566da6c3b0 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3104496044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 msgr2=0x55566da85c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc005940 tx=0x7f0cbc0058b0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3104496044 shutdown_connections 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da75e60 0x55566da785e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6cd60 0x55566da75620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3104496044 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da6c3b0 0x55566da85c10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3104496044 >> 172.21.15.105:0/3104496044 conn(0x55566da66ba0 msgr2=0x55566da77ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3104496044 shutdown_connections 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.261+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3104496044 wait complete. 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 -- start start 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 Processor -- start 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 0x55566da76260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6cd60 0x55566da767a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1acc30 con 0x55566da75e60 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1ac560 con 0x55566da6c3b0 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f0ccc1a97a0 con 0x55566da6cd60 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cd3bc1740 10 client.0 _ll_register_callbacks cb 0x7f0cc409c3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47664/0 (socket says 172.21.15.105:47664) 2024-04-22T06:38:15.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3941236383 learned_addr learned my addr 172.21.15.105:0/3941236383 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 0x55566da76260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cc9d9b640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6cd60 0x55566da767a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6cd60 msgr2=0x55566da767a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6cd60 0x55566da767a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3941236383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 msgr2=0x55566da76260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 0x55566da76260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 -- 172.21.15.105:0/3941236383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55566da81120 con 0x55566da75e60 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 0x55566da76260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0ccad9d640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4098f50 tx=0x7f0cc409cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9fb3ec4b3d195c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.262+0000 7f0cb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4020070 con 0x55566da75e60 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f0cc4085ac0 con 0x55566da75e60 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cbc02cc90 con 0x55566da75e60 2024-04-22T06:38:15.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cc409c580 con 0x55566da75e60 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 10 client.15972 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55566da81120 con 0x55566da75e60 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cbc02cc90 con 0x55566da75e60 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cc409b5d0 con 0x55566da75e60 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 10 client.15972 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 10 client.15972 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 20 client.15972 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da82630 0x55566daf2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 4 client.15972 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 --> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55566da79ed0 con 0x55566da82630 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.263+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da82630 0x55566daf2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cb1ffb640 10 client.15972 ms_handle_connect on v2:172.21.15.177:6838/1381266768 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cca59c640 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da82630 0x55566daf2030 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=296d91570b0c360 server_cookie=91795dfe2830862f in_seq=0 out_seq=0 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cb1ffb640 1 -- 172.21.15.105:0/3941236383 <== mds.1 v2:172.21.15.177:6838/1381266768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55566da79ed0 con 0x55566da82630 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cb1ffb640 10 client.15972 handle_command_reply: tid=0 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 client.15972 shutdown 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 msgr2=0x55566da76ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4098f50 tx=0x7f0cc409cdb0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 shutdown_connections 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:6838/1381266768,v1:172.21.15.177:6839/1381266768] conn(0x55566da82630 0x55566daf2030 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55566da75e60 0x55566da76ce0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55566da6cd60 0x55566da767a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cb1ffb640 0 client.15972 ms_handle_reset on v2:172.21.15.177:6838/1381266768 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 --2- 172.21.15.105:0/3941236383 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55566da6c3b0 0x55566da76260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 >> 172.21.15.105:0/3941236383 conn(0x55566da66ba0 msgr2=0x55566da65370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 shutdown_connections 2024-04-22T06:38:15.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 1 -- 172.21.15.105:0/3941236383 wait complete. 2024-04-22T06:38:15.272 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.264+0000 7f0cd3bc1740 20 client.15972 trim_cache size 0 max 16384 2024-04-22T06:38:15.272 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 msgr2=0x7f0cb40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 0x7f0cb40a6390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00058c0 tx=0x7f0cc00138f0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 msgr2=0x7f0ccc19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009fd0 tx=0x7f0cc400def0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 shutdown_connections 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f0cb40a3ed0 0x7f0cb40a6390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ccc107030 0x7f0ccc19cfa0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ccc101b40 0x7f0ccc198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 --2- 172.21.15.105:0/3619078040 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0ccc1011a0 0x7f0ccc1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.274 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 >> 172.21.15.105:0/3619078040 conn(0x7f0ccc076c50 msgr2=0x7f0ccc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.274 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 shutdown_connections 2024-04-22T06:38:15.274 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.265+0000 7f0cd0ef2640 1 -- 172.21.15.105:0/3619078040 wait complete. 2024-04-22T06:38:15.274 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:15 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2179631485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:15.275 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:38:15.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 -- 172.21.15.105:0/2274596524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f101010a0 msgr2=0x7f6f10101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 --2- 172.21.15.105:0/2274596524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f101010a0 0x7f6f10101480 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009900 tx=0x7f6f0c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 -- 172.21.15.105:0/2274596524 shutdown_connections 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 --2- 172.21.15.105:0/2274596524 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f10106170 0x7f6f1010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 --2- 172.21.15.105:0/2274596524 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f10101a50 0x7f6f10105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 --2- 172.21.15.105:0/2274596524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f101010a0 0x7f6f10101480 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 -- 172.21.15.105:0/2274596524 >> 172.21.15.105:0/2274596524 conn(0x7f6f10077eb0 msgr2=0x7f6f100782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 -- 172.21.15.105:0/2274596524 shutdown_connections 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.377+0000 7f6f18269640 1 -- 172.21.15.105:0/2274596524 wait complete. 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 Processor -- start 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 -- start start 2024-04-22T06:38:15.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f101010a0 0x7f6f10197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f1019be30 0x7f6f1019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101110b0 con 0x7f6f10106170 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10110f30 con 0x7f6f101010a0 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f18269640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10111230 con 0x7f6f1019be30 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47682/0 (socket says 172.21.15.105:47682) 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 -- 172.21.15.105:0/196463349 learned_addr learned my addr 172.21.15.105:0/196463349 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f16005640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f101010a0 0x7f6f10197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f16806640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f1019be30 0x7f6f1019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 -- 172.21.15.105:0/196463349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f1019be30 msgr2=0x7f6f1019c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f1019be30 0x7f6f1019c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 -- 172.21.15.105:0/196463349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f101010a0 msgr2=0x7f6f10197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f101010a0 0x7f6f10197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.378+0000 7f6f15804640 1 -- 172.21.15.105:0/196463349 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1019c920 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f15804640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c970 tx=0x7f6f0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f077fe640 1 -- 172.21.15.105:0/196463349 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00014070 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f077fe640 1 -- 172.21.15.105:0/196463349 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f00004440 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1019cc10 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f077fe640 1 -- 172.21.15.105:0/196463349 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00003e50 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.379+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f101a4450 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.381+0000 7f6f077fe640 1 -- 172.21.15.105:0/196463349 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f6f00017070 con 0x7f6f10106170 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.381+0000 7f6f1ae57740 1 Processor -- start 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 -- start start 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fde4d0 0x563eb0fde8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563eb0f431f0 con 0x563eb0fe7a00 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563eb0ee04b0 con 0x563eb0fde4d0 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563eb0f3f1b0 con 0x563eb0fdee80 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f077fe640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 0x7f6efc0a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f16806640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 0x7f6efc0a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f16005640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47690/0 (socket says 172.21.15.105:47690) 2024-04-22T06:38:15.394 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 -- 172.21.15.105:0/1063971533 learned_addr learned my addr 172.21.15.105:0/1063971533 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f077fe640 1 -- 172.21.15.105:0/196463349 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f6f000d0580 con 0x7f6f10106170 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f16806640 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fde4d0 0x563eb0fde8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 -- 172.21.15.105:0/1063971533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 msgr2=0x563eb0fe7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 -- 172.21.15.105:0/1063971533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fde4d0 msgr2=0x563eb0fde8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fde4d0 0x563eb0fde8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f15804640 1 -- 172.21.15.105:0/1063971533 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb0ed21b0 con 0x563eb0fe7a00 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.382+0000 7f6f16005640 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f16806640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 0x7f6efc0a65b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009270 tx=0x7f6f08012b10 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f15804640 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6f0009c050 tx=0x7f6f00000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1139ac7780c07d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f04bfe640 1 -- 172.21.15.105:0/1063971533 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00064f10 con 0x563eb0fe7a00 2024-04-22T06:38:15.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f04bfe640 1 -- 172.21.15.105:0/1063971533 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0009ca50 con 0x563eb0fe7a00 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f04bfe640 1 -- 172.21.15.105:0/1063971533 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0004b0b0 con 0x563eb0fe7a00 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1063971533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 msgr2=0x563eb0fe9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6f0009c050 tx=0x7f6f00000bc0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1063971533 shutdown_connections 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fe7a00 0x563eb0fe9e50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1063971533 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fde4d0 0x563eb0fde8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.383+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1063971533 >> 172.21.15.105:0/1063971533 conn(0x563eb0fd8ba0 msgr2=0x563eb0fd8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1063971533 shutdown_connections 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1063971533 wait complete. 2024-04-22T06:38:15.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- start start 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 Processor -- start 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fde4d0 0x563eb0fe7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563eb0fea970 con 0x563eb0fdee80 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563eb0efc500 con 0x563eb0fe7a00 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563eb0feaaf0 con 0x563eb0fde4d0 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f1ae57740 10 client.0 _ll_register_callbacks cb 0x563eb0ecab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47698/0 (socket says 172.21.15.105:47698) 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 -- 172.21.15.105:0/3146442577 learned_addr learned my addr 172.21.15.105:0/3146442577 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f16806640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 -- 172.21.15.105:0/3146442577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fde4d0 msgr2=0x563eb0fe7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fde4d0 0x563eb0fe7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 -- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 msgr2=0x563eb0fe4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.384+0000 7f6f15804640 1 -- 172.21.15.105:0/3146442577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb0c553f0 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f15804640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6f000cdc40 tx=0x7f6f000a0770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985ad3f184b84021 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0009c7b0 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f6f000a4b30 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f0009ca50 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f0009c350 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 10 client.15984 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563eb0d1f7d0 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0009ca50 con 0x563eb0fdee80 2024-04-22T06:38:15.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f0009b810 con 0x563eb0fdee80 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 10 client.15984 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 20 client.15984 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb0ff7ac0 0x563eb0ff9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 4 client.15984 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.385+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eb0ee2930 con 0x563eb0ff7ac0 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.386+0000 7f6f16005640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb0ff7ac0 0x563eb0ff9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.386+0000 7f6ef9ffb640 10 client.15984 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.386+0000 7f6f16005640 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb0ff7ac0 0x563eb0ff9eb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a13487c07aca04 server_cookie=dda0de81626009c2 in_seq=0 out_seq=0 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/3146442577 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563eb0ee2930 con 0x563eb0ff7ac0 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6ef9ffb640 10 client.15984 handle_command_reply: tid=0 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 client.15984 shutdown 2024-04-22T06:38:15.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 msgr2=0x563eb0fe46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6f000cdc40 tx=0x7f6f000a0770 comp rx=0 tx=0).stop 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 shutdown_connections 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb0ff7ac0 0x563eb0ff9eb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fdee80 0x563eb0fe46d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/3146442577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fde4d0 0x563eb0fe7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6ef9ffb640 0 client.15984 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 >> 172.21.15.105:0/3146442577 conn(0x563eb0fd8ba0 msgr2=0x563eb0fda8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 shutdown_connections 2024-04-22T06:38:15.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 1 -- 172.21.15.105:0/3146442577 wait complete. 2024-04-22T06:38:15.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.387+0000 7f6f1ae57740 20 client.15984 trim_cache size 0 max 16384 2024-04-22T06:38:15.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 Processor -- start 2024-04-22T06:38:15.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 -- start start 2024-04-22T06:38:15.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101ab730 con 0x563eb0fde4d0 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101ad270 con 0x563eb0fe7a00 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101abe00 con 0x563eb0fdee80 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f16005640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f16005640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47714/0 (socket says 172.21.15.105:47714) 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.400+0000 7f6f16005640 1 -- 172.21.15.105:0/1778494021 learned_addr learned my addr 172.21.15.105:0/1778494021 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16806640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f15804640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 -- 172.21.15.105:0/1778494021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 msgr2=0x563eb0fe3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 -- 172.21.15.105:0/1778494021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 msgr2=0x563eb0fe6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 -- 172.21.15.105:0/1778494021 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb0fea7c0 con 0x563eb0fde4d0 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16806640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f15804640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f16005640 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009900 tx=0x7f6f0c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307041c9721dd5bb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6efbfff640 1 -- 172.21.15.105:0/1778494021 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0043d0 con 0x563eb0fde4d0 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6efbfff640 1 -- 172.21.15.105:0/1778494021 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0c02cc90 con 0x563eb0fde4d0 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6efbfff640 1 -- 172.21.15.105:0/1778494021 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0046c0 con 0x563eb0fde4d0 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1778494021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 msgr2=0x563eb0ff3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009900 tx=0x7f6f0c009560 comp rx=0 tx=0).stop 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1778494021 shutdown_connections 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0fe6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0fe3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/1778494021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff3830 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1778494021 >> 172.21.15.105:0/1778494021 conn(0x563eb0fd8ba0 msgr2=0x563eb0fda890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1778494021 shutdown_connections 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.401+0000 7f6f1ae57740 1 -- 172.21.15.105:0/1778494021 wait complete. 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 -- start start 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 Processor -- start 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0feb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0feb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101acba0 con 0x563eb0fde4d0 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101ac4d0 con 0x563eb0fe7a00 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f6f101a9710 con 0x563eb0fdee80 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f1ae57740 10 client.0 _ll_register_callbacks cb 0x7f6f00065100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47726/0 (socket says 172.21.15.105:47726) 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 -- 172.21.15.105:0/2135797264 learned_addr learned my addr 172.21.15.105:0/2135797264 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:15.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f15804640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0feb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 -- 172.21.15.105:0/2135797264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 msgr2=0x563eb0feb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0feb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 -- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 msgr2=0x563eb0feb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16806640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0feb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0feb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f16005640 1 -- 172.21.15.105:0/2135797264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb0fec120 con 0x563eb0fde4d0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.402+0000 7f6f15804640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0feb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f16005640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c04e040 tx=0x7f6f0c004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b7fe36ba48b880 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c05b060 con 0x563eb0fde4d0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f6f0c0035f0 con 0x563eb0fde4d0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f0c02cc90 con 0x563eb0fde4d0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f0c02caf0 con 0x563eb0fde4d0 2024-04-22T06:38:15.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 10 client.15990 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563eb10620f0 con 0x563eb0fde4d0 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f0c02cc90 con 0x563eb0fde4d0 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f0c05e030 con 0x563eb0fde4d0 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 10 client.15990 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 20 client.15990 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:15.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb1062870 0x563eb1064c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 4 client.15990 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563eb0f50c10 con 0x563eb1062870 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.403+0000 7f6f15804640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb1062870 0x563eb1064c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6ef9ffb640 10 client.15990 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f15804640 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb1062870 0x563eb1064c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a8053abf3fbd61 server_cookie=9c7b2c29c336ae32 in_seq=0 out_seq=0 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6ef9ffb640 1 -- 172.21.15.105:0/2135797264 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563eb0f50c10 con 0x563eb1062870 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6ef9ffb640 10 client.15990 handle_command_reply: tid=0 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 client.15990 shutdown 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 msgr2=0x563eb0ff0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c04e040 tx=0x7f6f0c004540 comp rx=0 tx=0).stop 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 shutdown_connections 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563eb1062870 0x563eb1064c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563eb0fe7a00 0x563eb0feb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563eb0fdee80 0x563eb0feb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6ef9ffb640 0 client.15990 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 --2- 172.21.15.105:0/2135797264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563eb0fde4d0 0x563eb0ff0090 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 >> 172.21.15.105:0/2135797264 conn(0x563eb0fd8ba0 msgr2=0x563eb0fe3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 shutdown_connections 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 1 -- 172.21.15.105:0/2135797264 wait complete. 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.404+0000 7f6f1ae57740 20 client.15990 trim_cache size 0 max 16384 2024-04-22T06:38:15.417 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:15.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 msgr2=0x7f6efc0a65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 0x7f6efc0a65b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009270 tx=0x7f6f08012b10 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 msgr2=0x7f6f10197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c970 tx=0x7f6f0000ce30 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 shutdown_connections 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f6efc0a40f0 0x7f6efc0a65b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f1019be30 0x7f6f1019c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f10106170 0x7f6f10197ab0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 --2- 172.21.15.105:0/196463349 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6f101010a0 0x7f6f10197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 >> 172.21.15.105:0/196463349 conn(0x7f6f10077eb0 msgr2=0x7f6f10100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 shutdown_connections 2024-04-22T06:38:15.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:15.405+0000 7f6f18269640 1 -- 172.21.15.105:0/196463349 wait complete. 2024-04-22T06:38:15.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:15 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2179631485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:15.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:15 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2179631485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:16.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:16 smithi105 ceph-mon[27916]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 33 op/s 2024-04-22T06:38:16.420 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:38:16.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:38:16.421 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:38:16.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:16 smithi112 ceph-mon[31236]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 33 op/s 2024-04-22T06:38:16.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:16 smithi177 ceph-mon[31346]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 33 op/s 2024-04-22T06:38:16.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 -- 172.21.15.105:0/2752550109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 msgr2=0x7f5534103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 --2- 172.21.15.105:0/2752550109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f5534103430 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5524009900 tx=0x7f552402bcd0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 -- 172.21.15.105:0/2752550109 shutdown_connections 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 --2- 172.21.15.105:0/2752550109 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f553410c570 0x7f553410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 --2- 172.21.15.105:0/2752550109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5534103970 0x7f553410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 --2- 172.21.15.105:0/2752550109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f5534103430 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.536+0000 7f553b870640 1 -- 172.21.15.105:0/2752550109 >> 172.21.15.105:0/2752550109 conn(0x7f5534077fb0 msgr2=0x7f55340783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- 172.21.15.105:0/2752550109 shutdown_connections 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- 172.21.15.105:0/2752550109 wait complete. 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 Processor -- start 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- start start 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5534103970 0x7f553419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 0x7f55341985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5534112200 con 0x7f5534103050 2024-04-22T06:38:16.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5534112080 con 0x7f553410c570 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553b870640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5534112380 con 0x7f5534103970 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553960c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553960c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47736/0 (socket says 172.21.15.105:47736) 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.537+0000 7f553960c640 1 -- 172.21.15.105:0/4191359685 learned_addr learned my addr 172.21.15.105:0/4191359685 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f5538e0b640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5534103970 0x7f553419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f5539e0d640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 0x7f55341985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 -- 172.21.15.105:0/4191359685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5534103970 msgr2=0x7f553419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5534103970 0x7f553419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 -- 172.21.15.105:0/4191359685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 msgr2=0x7f55341985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 0x7f55341985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 -- 172.21.15.105:0/4191359685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5534198d40 con 0x7f5534103050 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f5539e0d640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 0x7f55341985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553960c640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5524005940 tx=0x7f552402d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f552a7fc640 1 -- 172.21.15.105:0/4191359685 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524043020 con 0x7f5534103050 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f552a7fc640 1 -- 172.21.15.105:0/4191359685 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f552402dd10 con 0x7f5534103050 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f552a7fc640 1 -- 172.21.15.105:0/4191359685 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524003b80 con 0x7f5534103050 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5534198fd0 con 0x7f5534103050 2024-04-22T06:38:16.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.538+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55341a52d0 con 0x7f5534103050 2024-04-22T06:38:16.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f552a7fc640 1 -- 172.21.15.105:0/4191359685 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f5524046020 con 0x7f5534103050 2024-04-22T06:38:16.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 Processor -- start 2024-04-22T06:38:16.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 -- start start 2024-04-22T06:38:16.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 0x55e175e3d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e40260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e175d98f00 con 0x55e175e348e0 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e175d36590 con 0x55e175e35290 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f553e45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e175d94ec0 con 0x55e175e3de10 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f552a7fc640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 0x7f55200a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f5539e0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.541+0000 7f5539e0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47740/0 (socket says 172.21.15.105:47740) 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 -- 172.21.15.105:0/230355760 learned_addr learned my addr 172.21.15.105:0/230355760 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 0x7f55200a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5538e0b640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e40260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f552a7fc640 1 -- 172.21.15.105:0/4191359685 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f55240f2b90 con 0x7f5534103050 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f553960c640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 0x55e175e3d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 -- 172.21.15.105:0/230355760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 msgr2=0x55e175e40260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e40260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 -- 172.21.15.105:0/230355760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 msgr2=0x55e175e3d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 0x55e175e3d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 -- 172.21.15.105:0/230355760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e175b737e0 con 0x55e175e348e0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f553960c640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 0x55e175e3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55300167b0 tx=0x7f5530016c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3caf68e94d64918b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f5539e0d640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 0x7f55200a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5530014720 tx=0x7f553001a000 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f551f7fe640 1 -- 172.21.15.105:0/230355760 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530024070 con 0x55e175e348e0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f551f7fe640 1 -- 172.21.15.105:0/230355760 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f553000a840 con 0x55e175e348e0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.542+0000 7f551f7fe640 1 -- 172.21.15.105:0/230355760 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553001f650 con 0x55e175e348e0 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- 172.21.15.105:0/230355760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 msgr2=0x55e175e34cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55300167b0 tx=0x7f5530016c70 comp rx=0 tx=0).stop 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- 172.21.15.105:0/230355760 shutdown_connections 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e40260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e35290 0x55e175e3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 --2- 172.21.15.105:0/230355760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e34cc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- 172.21.15.105:0/230355760 >> 172.21.15.105:0/230355760 conn(0x55e175e2efb0 msgr2=0x55e175e2f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- 172.21.15.105:0/230355760 shutdown_connections 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- 172.21.15.105:0/230355760 wait complete. 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 -- start start 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.543+0000 7f553e45b740 1 Processor -- start 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e3ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e3b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e175e40d20 con 0x55e175e35290 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e175d52580 con 0x55e175e348e0 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e175e40ea0 con 0x55e175e3de10 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553e45b740 10 client.0 _ll_register_callbacks cb 0x55e175d58650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47744/0 (socket says 172.21.15.105:47744) 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 -- 172.21.15.105:0/3646918002 learned_addr learned my addr 172.21.15.105:0/3646918002 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5539e0d640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e3b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f553960c640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e3ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 -- 172.21.15.105:0/3646918002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 msgr2=0x55e175e3b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e3b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 -- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 msgr2=0x55e175e3ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e3ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 -- 172.21.15.105:0/3646918002 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e175bd3f70 con 0x55e175e35290 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5539e0d640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e3b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:16.551 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f5538e0b640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f552c010ac0 tx=0x7f552c010f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f81ad6b373bee9c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.544+0000 7f551cff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c00fe30 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f552c004b20 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f553000a840 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f552c004d20 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 10 client.16002 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e175a6b170 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f553000a840 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f552c019410 con 0x55e175e35290 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 10 client.16002 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 20 client.16002 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:16.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e4e2d0 0x55e175e50710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 4 client.16002 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e175da6920 con 0x55e175e4e2d0 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.545+0000 7f553960c640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e4e2d0 0x55e175e50710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.546+0000 7f553960c640 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e4e2d0 0x55e175e50710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19be956b778d8fa2 server_cookie=40da711a070648a3 in_seq=0 out_seq=0 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.546+0000 7f551cff9640 10 client.16002 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.546+0000 7f551cff9640 1 -- 172.21.15.105:0/3646918002 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e175da6920 con 0x55e175e4e2d0 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.546+0000 7f551cff9640 10 client.16002 handle_command_reply: tid=0 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.546+0000 7f553e45b740 1 client.16002 shutdown 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 msgr2=0x55e175e3b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f552c010ac0 tx=0x7f552c010f80 comp rx=0 tx=0).stop 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 shutdown_connections 2024-04-22T06:38:16.553 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e4e2d0 0x55e175e50710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e3de10 0x55e175e3b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e3b2b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f551cff9640 0 client.16002 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 --2- 172.21.15.105:0/3646918002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e3ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 >> 172.21.15.105:0/3646918002 conn(0x55e175e2efb0 msgr2=0x55e175e30d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 shutdown_connections 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 1 -- 172.21.15.105:0/3646918002 wait complete. 2024-04-22T06:38:16.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.547+0000 7f553e45b740 20 client.16002 trim_cache size 0 max 16384 2024-04-22T06:38:16.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 Processor -- start 2024-04-22T06:38:16.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 -- start start 2024-04-22T06:38:16.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e2efb0 0x55e175e3d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e35290 0x55e175e55f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55341ada70 con 0x55e175e348e0 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55341af5b0 con 0x55e175e2efb0 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f553e45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55341ae140 con 0x55e175e35290 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.564+0000 7f5538e0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47746/0 (socket says 172.21.15.105:47746) 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 -- 172.21.15.105:0/2285183103 learned_addr learned my addr 172.21.15.105:0/2285183103 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 -- 172.21.15.105:0/2285183103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e35290 msgr2=0x55e175e55f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e35290 0x55e175e55f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 -- 172.21.15.105:0/2285183103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e2efb0 msgr2=0x55e175e3d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e2efb0 0x55e175e3d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 -- 172.21.15.105:0/2285183103 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e175e2eae0 con 0x55e175e348e0 2024-04-22T06:38:16.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f5538e0b640 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f552c010a90 tx=0x7f552c00f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7e5bde838c7d14 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f551effd640 1 -- 172.21.15.105:0/2285183103 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c00fe30 con 0x55e175e348e0 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f551effd640 1 -- 172.21.15.105:0/2285183103 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f552c0049e0 con 0x55e175e348e0 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.565+0000 7f551effd640 1 -- 172.21.15.105:0/2285183103 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c02a810 con 0x55e175e348e0 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- 172.21.15.105:0/2285183103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 msgr2=0x55e175e49be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f552c010a90 tx=0x7f552c00f8b0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- 172.21.15.105:0/2285183103 shutdown_connections 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e35290 0x55e175e55f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e348e0 0x55e175e49be0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- 172.21.15.105:0/2285183103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e2efb0 0x55e175e3d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- 172.21.15.105:0/2285183103 >> 172.21.15.105:0/2285183103 conn(0x55e175e359e0 msgr2=0x55e175e3b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- 172.21.15.105:0/2285183103 shutdown_connections 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- 172.21.15.105:0/2285183103 wait complete. 2024-04-22T06:38:16.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- start start 2024-04-22T06:38:16.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 Processor -- start 2024-04-22T06:38:16.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e2efb0 0x55e175e462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.570 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.566+0000 7f553e45b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55341aeee0 con 0x55e175e35290 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55341ae810 con 0x55e175e348e0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55341aba50 con 0x55e175e2efb0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 10 client.0 _ll_register_callbacks cb 0x7f552c0106d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47756/0 (socket says 172.21.15.105:47756) 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 -- 172.21.15.105:0/79259094 learned_addr learned my addr 172.21.15.105:0/79259094 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553960c640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e2efb0 0x55e175e462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5538e0b640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 -- 172.21.15.105:0/79259094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e2efb0 msgr2=0x55e175e462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e2efb0 0x55e175e462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 -- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 msgr2=0x55e175e430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 -- 172.21.15.105:0/79259094 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e175e43b40 con 0x55e175e35290 2024-04-22T06:38:16.571 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f5539e0d640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55300167b0 tx=0x7f553000ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab148340f3330b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f551cff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530024070 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f5530029440 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f552c0049e0 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5530037050 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 10 client.16008 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.567+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e175eb55a0 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f552c0049e0 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f553001e030 con 0x55e175e35290 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 10 client.16008 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 20 client.16008 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e3de10 0x55e175eb7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 4 client.16008 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e175da0790 con 0x55e175e3de10 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553960c640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e3de10 0x55e175eb7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f551cff9640 10 client.16008 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.568+0000 7f553960c640 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e3de10 0x55e175eb7b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe4dc5f7ceccf8ad server_cookie=892d956dfaa6d40d in_seq=0 out_seq=0 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.569+0000 7f551cff9640 1 -- 172.21.15.105:0/79259094 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e175da0790 con 0x55e175e3de10 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.569+0000 7f551cff9640 10 client.16008 handle_command_reply: tid=0 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.569+0000 7f553e45b740 1 client.16008 shutdown 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 msgr2=0x55e175e43600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f55300167b0 tx=0x7f553000ad30 comp rx=0 tx=0).stop 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 shutdown_connections 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e175e3de10 0x55e175eb7b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e175e35290 0x55e175e43600 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e175e348e0 0x55e175e430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f551cff9640 0 client.16008 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 --2- 172.21.15.105:0/79259094 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e175e2efb0 0x55e175e462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 >> 172.21.15.105:0/79259094 conn(0x55e175e359e0 msgr2=0x55e175e2d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 shutdown_connections 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 1 -- 172.21.15.105:0/79259094 wait complete. 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553e45b740 20 client.16008 trim_cache size 0 max 16384 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stdout: "return_code": 0, 2024-04-22T06:38:16.574 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_tag": "d076cafc-5bf0-4c4d-b0c8-8dc7bc4a1916", 2024-04-22T06:38:16.575 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": "asynchronous" 2024-04-22T06:38:16.575 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:16.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 msgr2=0x7f55200a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 0x7f55200a6390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5530014720 tx=0x7f553001a000 comp rx=0 tx=0).stop 2024-04-22T06:38:16.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 msgr2=0x7f553419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5524005940 tx=0x7f552402d910 comp rx=0 tx=0).stop 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 shutdown_connections 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55200a3ed0 0x7f55200a6390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f553410c570 0x7f55341985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5534103970 0x7f553419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 --2- 172.21.15.105:0/4191359685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5534103050 0x7f553419e410 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.570+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 >> 172.21.15.105:0/4191359685 conn(0x7f5534077fb0 msgr2=0x7f55340fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.571+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 shutdown_connections 2024-04-22T06:38:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.571+0000 7f553b870640 1 -- 172.21.15.105:0/4191359685 wait complete. 2024-04-22T06:38:16.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:38:16.580 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 -- 172.21.15.105:0/1269197934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9130103aa0 msgr2=0x7f913010bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 --2- 172.21.15.105:0/1269197934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9130103aa0 0x7f913010bf80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9120009900 tx=0x7f912002bcd0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 -- 172.21.15.105:0/1269197934 shutdown_connections 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 --2- 172.21.15.105:0/1269197934 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f913010c550 0x7f913010e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 --2- 172.21.15.105:0/1269197934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9130103aa0 0x7f913010bf80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.683 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 --2- 172.21.15.105:0/1269197934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103180 0x7f9130103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 -- 172.21.15.105:0/1269197934 >> 172.21.15.105:0/1269197934 conn(0x7f91300771d0 msgr2=0x7f91300775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 -- 172.21.15.105:0/1269197934 shutdown_connections 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.682+0000 7f9137e6d640 1 -- 172.21.15.105:0/1269197934 wait complete. 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 Processor -- start 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 -- start start 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 0x7f9130198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103aa0 0x7f9130198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f91301121f0 con 0x7f913010c550 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9130112070 con 0x7f9130103180 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9137e6d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9130112370 con 0x7f9130103aa0 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47776/0 (socket says 172.21.15.105:47776) 2024-04-22T06:38:16.684 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 -- 172.21.15.105:0/3253997542 learned_addr learned my addr 172.21.15.105:0/3253997542 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9135c09640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 0x7f9130198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9135408640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103aa0 0x7f9130198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 -- 172.21.15.105:0/3253997542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103aa0 msgr2=0x7f9130198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103aa0 0x7f9130198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 -- 172.21.15.105:0/3253997542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 msgr2=0x7f9130198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 0x7f9130198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 -- 172.21.15.105:0/3253997542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913019d740 con 0x7f913010c550 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f9135c09640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 0x7f9130198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.683+0000 7f913640a640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f912800ccb0 tx=0x7f912800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.684+0000 7f9126ffd640 1 -- 172.21.15.105:0/3253997542 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912800f450 con 0x7f913010c550 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.684+0000 7f9126ffd640 1 -- 172.21.15.105:0/3253997542 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f912800faa0 con 0x7f913010c550 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.684+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913019da30 con 0x7f913010c550 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.684+0000 7f9126ffd640 1 -- 172.21.15.105:0/3253997542 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128018b00 con 0x7f913010c550 2024-04-22T06:38:16.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.684+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91301a52c0 con 0x7f913010c550 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f9126ffd640 1 -- 172.21.15.105:0/3253997542 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f912800fc50 con 0x7f913010c550 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 Processor -- start 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 -- start start 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bab4b851e0 con 0x55bab4c29a80 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bab4b22390 con 0x55bab4c20ea0 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.686+0000 7f913aa65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bab4b811a0 con 0x55bab4c20580 2024-04-22T06:38:16.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47784/0 (socket says 172.21.15.105:47784) 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 -- 172.21.15.105:0/2202477939 learned_addr learned my addr 172.21.15.105:0/2202477939 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135c09640 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9126ffd640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 0x7f911c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 -- 172.21.15.105:0/2202477939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 msgr2=0x55bab4c20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 -- 172.21.15.105:0/2202477939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 msgr2=0x55bab4c29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 -- 172.21.15.105:0/2202477939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bab49617d0 con 0x55bab4c29a80 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f913640a640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 0x7f911c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9126ffd640 1 -- 172.21.15.105:0/3253997542 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f91280d1200 con 0x7f913010c550 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f9135408640 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f912002c210 tx=0x7f9120005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15af3b84f049ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f911bfff640 1 -- 172.21.15.105:0/2202477939 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120048070 con 0x55bab4c29a80 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f911bfff640 1 -- 172.21.15.105:0/2202477939 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f912002cd30 con 0x55bab4c29a80 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.687+0000 7f911bfff640 1 -- 172.21.15.105:0/2202477939 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120043440 con 0x55bab4c29a80 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 -- 172.21.15.105:0/2202477939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 msgr2=0x55bab4c2be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913640a640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 0x7f911c0a6410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f91280ce820 tx=0x7f912806ea50 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f912002c210 tx=0x7f9120005e30 comp rx=0 tx=0).stop 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 -- 172.21.15.105:0/2202477939 shutdown_connections 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c2be70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 --2- 172.21.15.105:0/2202477939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 -- 172.21.15.105:0/2202477939 >> 172.21.15.105:0/2202477939 conn(0x55bab4c1acb0 msgr2=0x55bab4c1b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 -- 172.21.15.105:0/2202477939 shutdown_connections 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.688+0000 7f913aa65740 1 -- 172.21.15.105:0/2202477939 wait complete. 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 -- start start 2024-04-22T06:38:16.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 Processor -- start 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c26b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c29a80 0x55bab4c27590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bab4c2c810 con 0x55bab4c20ea0 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bab4b3e2f0 con 0x55bab4c29a80 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bab4c2c990 con 0x55bab4c20580 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f913aa65740 10 client.0 _ll_register_callbacks cb 0x55bab4b4b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47796/0 (socket says 172.21.15.105:47796) 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 -- 172.21.15.105:0/2978515747 learned_addr learned my addr 172.21.15.105:0/2978515747 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135408640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c29a80 0x55bab4c27590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135408640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c26b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 -- 172.21.15.105:0/2978515747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 msgr2=0x55bab4c26b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c26b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 -- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c29a80 msgr2=0x55bab4c27590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c29a80 0x55bab4c27590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.689+0000 7f9135c09640 1 -- 172.21.15.105:0/2978515747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bab48973f0 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f9135c09640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f912c010500 tx=0x7f912c0109c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25db119b0b5611ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c018070 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f912c014640 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f912002cd30 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f912c024050 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 10 client.16020 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bab48570f0 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f912002cd30 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f912c012980 con 0x55bab4c20ea0 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 10 client.16020 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:16.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 20 client.16020 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4c39b70 0x55bab4c3bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 4 client.16020 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.690+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bab4b92c00 con 0x55bab4c39b70 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.691+0000 7f9135408640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4c39b70 0x55bab4c3bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.691+0000 7f91197fa640 10 client.16020 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.691+0000 7f9135408640 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4c39b70 0x55bab4c3bfb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fddd54f524780601 server_cookie=73d577b137f62cdf in_seq=0 out_seq=0 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f91197fa640 1 -- 172.21.15.105:0/2978515747 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bab4b92c00 con 0x55bab4c39b70 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f91197fa640 10 client.16020 handle_command_reply: tid=0 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 client.16020 shutdown 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 msgr2=0x55bab4c27050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f912c010500 tx=0x7f912c0109c0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 shutdown_connections 2024-04-22T06:38:16.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4c39b70 0x55bab4c3bfb0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c29a80 0x55bab4c27590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c20ea0 0x55bab4c27050 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f91197fa640 0 client.16020 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 --2- 172.21.15.105:0/2978515747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c26b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 >> 172.21.15.105:0/2978515747 conn(0x55bab4c1acb0 msgr2=0x55bab4c1ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 shutdown_connections 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 1 -- 172.21.15.105:0/2978515747 wait complete. 2024-04-22T06:38:16.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.692+0000 7f913aa65740 20 client.16020 trim_cache size 0 max 16384 2024-04-22T06:38:16.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 Processor -- start 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 -- start start 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20580 0x55bab4c35730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20ea0 0x55bab4c25c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f91301ad930 con 0x55bab4c29a80 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f91301af470 con 0x55bab4c20580 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.709+0000 7f913aa65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91301ae000 con 0x55bab4c20ea0 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47810/0 (socket says 172.21.15.105:47810) 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 -- 172.21.15.105:0/552749290 learned_addr learned my addr 172.21.15.105:0/552749290 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 -- 172.21.15.105:0/552749290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20ea0 msgr2=0x55bab4c25c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20ea0 0x55bab4c25c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 -- 172.21.15.105:0/552749290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20580 msgr2=0x55bab4c35730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20580 0x55bab4c35730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 -- 172.21.15.105:0/552749290 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bab4c25060 con 0x55bab4c29a80 2024-04-22T06:38:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f9135408640 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x55bab4c1be40 tx=0x7f9120003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c751aec46a243b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f911b7fe640 1 -- 172.21.15.105:0/552749290 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120059070 con 0x55bab4c29a80 2024-04-22T06:38:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f911b7fe640 1 -- 172.21.15.105:0/552749290 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9120049040 con 0x55bab4c29a80 2024-04-22T06:38:16.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.710+0000 7f911b7fe640 1 -- 172.21.15.105:0/552749290 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120048040 con 0x55bab4c29a80 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- 172.21.15.105:0/552749290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 msgr2=0x55bab4c273b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x55bab4c1be40 tx=0x7f9120003c00 comp rx=0 tx=0).stop 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- 172.21.15.105:0/552749290 shutdown_connections 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c273b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20ea0 0x55bab4c25c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.714 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 --2- 172.21.15.105:0/552749290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20580 0x55bab4c35730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- 172.21.15.105:0/552749290 >> 172.21.15.105:0/552749290 conn(0x55bab4c1acb0 msgr2=0x55bab4c194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- 172.21.15.105:0/552749290 shutdown_connections 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- 172.21.15.105:0/552749290 wait complete. 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 -- start start 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.711+0000 7f913aa65740 1 Processor -- start 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c31f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c23460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f91301aeda0 con 0x55bab4c29a80 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f91301ae6d0 con 0x55bab4c20ea0 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f91301ab930 con 0x55bab4c20580 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 10 client.0 _ll_register_callbacks cb 0x7f912c007520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47822/0 (socket says 172.21.15.105:47822) 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 -- 172.21.15.105:0/1734034287 learned_addr learned my addr 172.21.15.105:0/1734034287 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c31f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135c09640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c23460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 -- 172.21.15.105:0/1734034287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 msgr2=0x55bab4c31f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c31f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 -- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 msgr2=0x55bab4c23460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c23460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 -- 172.21.15.105:0/1734034287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bab4c24110 con 0x55bab4c29a80 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135c09640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c23460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f9135408640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9120057040 tx=0x7f9120004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2901625f92dde server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f91197fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120059040 con 0x55bab4c29a80 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f9120003860 con 0x55bab4c29a80 2024-04-22T06:38:16.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9120049040 con 0x55bab4c29a80 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9120048040 con 0x55bab4c29a80 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 10 client.16026 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.712+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bab4ca00a0 con 0x55bab4c29a80 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9120049040 con 0x55bab4c29a80 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9120068050 con 0x55bab4c29a80 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 10 client.16026 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 20 client.16026 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4ca02d0 0x55bab4ca26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 4 client.16026 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bab4b8ca70 con 0x55bab4ca02d0 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f9135c09640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4ca02d0 0x55bab4ca26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:16.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f91197fa640 10 client.16026 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.713+0000 7f9135c09640 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4ca02d0 0x55bab4ca26c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39a55cc1484492dd server_cookie=591b8f7adc66568 in_seq=0 out_seq=0 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f91197fa640 1 -- 172.21.15.105:0/1734034287 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bab4b8ca70 con 0x55bab4ca02d0 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f91197fa640 10 client.16026 handle_command_reply: tid=0 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 client.16026 shutdown 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 msgr2=0x55bab4c239a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9120057040 tx=0x7f9120004270 comp rx=0 tx=0).stop 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 shutdown_connections 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bab4ca02d0 0x55bab4ca26c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bab4c29a80 0x55bab4c239a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f91197fa640 0 client.16026 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bab4c20ea0 0x55bab4c23460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 --2- 172.21.15.105:0/1734034287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bab4c20580 0x55bab4c31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 >> 172.21.15.105:0/1734034287 conn(0x55bab4c1acb0 msgr2=0x55bab4c1c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.718 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 shutdown_connections 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 1 -- 172.21.15.105:0/1734034287 wait complete. 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f913aa65740 20 client.16026 trim_cache size 0 max 16384 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": { 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "d076cafc-5bf0-4c4d-b0c8-8dc7bc4a1916": { 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "path": "/", 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "tag": "d076cafc-5bf0-4c4d-b0c8-8dc7bc4a1916", 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: "options": "recursive,force" 2024-04-22T06:38:16.719 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:16.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:16.720 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:16.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 msgr2=0x7f911c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.720 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 0x7f911c0a6410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f91280ce820 tx=0x7f912806ea50 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 msgr2=0x7f913019d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f912800ccb0 tx=0x7f912800a620 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 shutdown_connections 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f911c0a3f50 0x7f911c0a6410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f913010c550 0x7f913019d010 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9130103aa0 0x7f9130198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.714+0000 7f9137e6d640 1 --2- 172.21.15.105:0/3253997542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9130103180 0x7f9130198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.715+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 >> 172.21.15.105:0/3253997542 conn(0x7f91300771d0 msgr2=0x7f91300fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.715+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 shutdown_connections 2024-04-22T06:38:16.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:16.715+0000 7f9137e6d640 1 -- 172.21.15.105:0/3253997542 wait complete. 2024-04-22T06:38:16.723 INFO:tasks.cephfs.filesystem:scrub status for tag:d076cafc-5bf0-4c4d-b0c8-8dc7bc4a1916 - {'path': '/', 'tag': 'd076cafc-5bf0-4c4d-b0c8-8dc7bc4a1916', 'options': 'recursive,force'} 2024-04-22T06:38:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:18 smithi105 ceph-mon[27916]: scrub queued for path: / 2024-04-22T06:38:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:18 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:18 smithi105 ceph-mon[27916]: scrub summary: active paths [/] 2024-04-22T06:38:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:18 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:18 smithi105 ceph-mon[27916]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-22T06:38:18.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:18 smithi112 ceph-mon[31236]: scrub queued for path: / 2024-04-22T06:38:18.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:18 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:18 smithi112 ceph-mon[31236]: scrub summary: active paths [/] 2024-04-22T06:38:18.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:18 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:18 smithi112 ceph-mon[31236]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-22T06:38:18.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:18 smithi177 ceph-mon[31346]: scrub queued for path: / 2024-04-22T06:38:18.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:18 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:18 smithi177 ceph-mon[31346]: scrub summary: active paths [/] 2024-04-22T06:38:18.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:18 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:18.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:18 smithi177 ceph-mon[31346]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-22T06:38:20.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:20 smithi105 ceph-mon[27916]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-04-22T06:38:20.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:20 smithi112 ceph-mon[31236]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-04-22T06:38:20.562 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:20 smithi177 ceph-mon[31346]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-04-22T06:38:21.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:21 smithi105 ceph-mon[27916]: scrub summary: idle 2024-04-22T06:38:21.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:21 smithi112 ceph-mon[31236]: scrub summary: idle 2024-04-22T06:38:21.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:21 smithi177 ceph-mon[31346]: scrub summary: idle 2024-04-22T06:38:22.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:22 smithi105 ceph-mon[27916]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-22T06:38:22.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:22 smithi112 ceph-mon[31236]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-22T06:38:22.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:22 smithi177 ceph-mon[31346]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-22T06:38:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-mon[27916]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-22T06:38:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-mon[31236]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-22T06:38:24.465 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:38:24.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-mon[31346]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-04-22T06:38:24.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2600258924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb30100e80 msgr2=0x7ffb3010ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2600258924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb30100e80 0x7ffb3010ede0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009cd0 tx=0x7ffb2c02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2600258924 shutdown_connections 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2600258924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb30100e80 0x7ffb3010ede0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2600258924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb301004c0 0x7ffb30100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2600258924 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30106f50 0x7ffb30107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.581 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2600258924 >> 172.21.15.105:0/2600258924 conn(0x7ffb30077450 msgr2=0x7ffb300fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2600258924 shutdown_connections 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.577+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2600258924 wait complete. 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 Processor -- start 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 -- start start 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30100e80 0x7ffb30198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb30106f50 0x7ffb3019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30112280 con 0x7ffb301004c0 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30112100 con 0x7ffb30100e80 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb37b2e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30112400 con 0x7ffb30106f50 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33296/0 (socket says 172.21.15.105:33296) 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 -- 172.21.15.105:0/2086374609 learned_addr learned my addr 172.21.15.105:0/2086374609 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb350c9640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30100e80 0x7ffb30198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 -- 172.21.15.105:0/2086374609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb30106f50 msgr2=0x7ffb3019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb30106f50 0x7ffb3019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 -- 172.21.15.105:0/2086374609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30100e80 msgr2=0x7ffb30198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30100e80 0x7ffb30198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.578+0000 7ffb358ca640 1 -- 172.21.15.105:0/2086374609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3019d730 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb358ca640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800d950 tx=0x7ffb2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28017070 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3019da20 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb28009c30 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800f620 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.579+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb301a52b0 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.580+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30069230 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.581+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7ffb28005030 con 0x7ffb301004c0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.581+0000 7ffb26ffd640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 0x7ffb1c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.582+0000 7ffb350c9640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 0x7ffb1c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.582+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7ffb280d0740 con 0x7ffb301004c0 2024-04-22T06:38:24.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.582+0000 7ffb350c9640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 0x7ffb1c0a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb20005fd0 tx=0x7ffb20005e20 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.584 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.585+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb2809cc10 con 0x7ffb301004c0 2024-04-22T06:38:24.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.802+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb3002d5c0 con 0x7ffb301004c0 2024-04-22T06:38:24.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.803+0000 7ffb26ffd640 1 -- 172.21.15.105:0/2086374609 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7ffb28009de0 con 0x7ffb301004c0 2024-04-22T06:38:24.802 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:38:24.802 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:37:35.576345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24472,"mds_2":24461,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.105:6841/2147123680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":2147123680},{"type":"v1","addr":"172.21.15.105:6841","nonce":2147123680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.177:6837/4114728122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":4114728122},{"type":"v1","addr":"172.21.15.177:6837","nonce":4114728122}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6837/3970644798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3970644798},{"type":"v1","addr":"172.21.15.112:6837","nonce":3970644798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.177:6839/1381266768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":1381266768},{"type":"v1","addr":"172.21.15.177:6839","nonce":1381266768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[14640,24454,24461,24472,24500]},"id":1}]} 2024-04-22T06:38:24.802 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 27 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 msgr2=0x7ffb1c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 0x7ffb1c0a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb20005fd0 tx=0x7ffb20005e20 comp rx=0 tx=0).stop 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 msgr2=0x7ffb30198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800d950 tx=0x7ffb2800de10 comp rx=0 tx=0).stop 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 shutdown_connections 2024-04-22T06:38:24.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ffb1c0a3ed0 0x7ffb1c0a6390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb30106f50 0x7ffb3019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffb30100e80 0x7ffb30198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 --2- 172.21.15.105:0/2086374609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffb301004c0 0x7ffb30198740 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 >> 172.21.15.105:0/2086374609 conn(0x7ffb30077450 msgr2=0x7ffb30104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 shutdown_connections 2024-04-22T06:38:24.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.805+0000 7ffb37b2e640 1 -- 172.21.15.105:0/2086374609 wait complete. 2024-04-22T06:38:24.815 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 2ad732f1 --release --await' 2024-04-22T06:38:24.815 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db --set-id 2ad732f1 --release --await 2024-04-22T06:38:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 -- 172.21.15.105:0/2981263396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc103b20 msgr2=0x7f99cc10c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 --2- 172.21.15.105:0/2981263396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc103b20 0x7f99cc10c000 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc02bcd0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 -- 172.21.15.105:0/2981263396 shutdown_connections 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 --2- 172.21.15.105:0/2981263396 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc10c5d0 0x7f99cc10ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 --2- 172.21.15.105:0/2981263396 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc103b20 0x7f99cc10c000 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 --2- 172.21.15.105:0/2981263396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 0x7f99cc1035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 -- 172.21.15.105:0/2981263396 >> 172.21.15.105:0/2981263396 conn(0x7f99cc0771d0 msgr2=0x7f99cc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 -- 172.21.15.105:0/2981263396 shutdown_connections 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 -- 172.21.15.105:0/2981263396 wait complete. 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.925+0000 7f99d4152640 1 Processor -- start 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 -- start start 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 0x7f99cc1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc103b20 0x7f99cc198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc112210 con 0x7f99cc10c5d0 2024-04-22T06:38:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc112090 con 0x7f99cc103200 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d4152640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc112390 con 0x7f99cc103b20 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33310/0 (socket says 172.21.15.105:33310) 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 -- 172.21.15.105:0/2801101274 learned_addr learned my addr 172.21.15.105:0/2801101274 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc103b20 0x7f99cc198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 0x7f99cc1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 -- 172.21.15.105:0/2801101274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc103b20 msgr2=0x7f99cc198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc103b20 0x7f99cc198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 -- 172.21.15.105:0/2801101274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 msgr2=0x7f99cc1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 0x7f99cc1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 -- 172.21.15.105:0/2801101274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc19d650 con 0x7f99cc10c5d0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.926+0000 7f99d26ef640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e970 tx=0x7f99c800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.927+0000 7f99c2ffd640 1 -- 172.21.15.105:0/2801101274 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c800ccc0 con 0x7f99cc10c5d0 2024-04-22T06:38:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.927+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc19d940 con 0x7f99cc10c5d0 2024-04-22T06:38:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.927+0000 7f99c2ffd640 1 -- 172.21.15.105:0/2801101274 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c80054d0 con 0x7f99cc10c5d0 2024-04-22T06:38:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.927+0000 7f99c2ffd640 1 -- 172.21.15.105:0/2801101274 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8010430 con 0x7f99cc10c5d0 2024-04-22T06:38:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.927+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc1a51d0 con 0x7f99cc10c5d0 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 -- 172.21.15.105:0/2732425647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4781031d0 msgr2=0x7fc4781035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 --2- 172.21.15.105:0/2732425647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4781031d0 0x7fc4781035b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc474009900 tx=0x7fc47402bd60 comp rx=0 tx=0).stop 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 -- 172.21.15.105:0/2732425647 shutdown_connections 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 --2- 172.21.15.105:0/2732425647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc47810c510 0x7fc47810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 --2- 172.21.15.105:0/2732425647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc478103af0 0x7fc47810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 --2- 172.21.15.105:0/2732425647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4781031d0 0x7fc4781035b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 -- 172.21.15.105:0/2732425647 >> 172.21.15.105:0/2732425647 conn(0x7fc478077f50 msgr2=0x7fc478078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 -- 172.21.15.105:0/2732425647 shutdown_connections 2024-04-22T06:38:24.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.929+0000 7fc47f515640 1 -- 172.21.15.105:0/2732425647 wait complete. 2024-04-22T06:38:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99c2ffd640 1 -- 172.21.15.105:0/2801101274 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f99c8005020 con 0x7f99cc10c5d0 2024-04-22T06:38:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 Processor -- start 2024-04-22T06:38:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 -- start start 2024-04-22T06:38:24.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 Processor -- start 2024-04-22T06:38:24.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 -- start start 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd876620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd8824d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5556dd7db8d0 con 0x5556dd876b60 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5556dd778e80 con 0x5556dd8800e0 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d6e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5556dd7d7890 con 0x5556dd8778d0 2024-04-22T06:38:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d1eee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 0x7fc478100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d1eee640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33328/0 (socket says 172.21.15.105:33328) 2024-04-22T06:38:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d1eee640 1 -- 172.21.15.105:0/2839020966 learned_addr learned my addr 172.21.15.105:0/2839020966 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc47810c510 0x7fc4781018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.932 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781121a0 con 0x7fc478103af0 2024-04-22T06:38:24.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc478112020 con 0x7fc47810c510 2024-04-22T06:38:24.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47f515640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc478112320 con 0x7fc4781031d0 2024-04-22T06:38:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99c2ffd640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 0x7f99b80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.930+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd8824d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.930+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.933 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33344/0 (socket says 172.21.15.105:33344) 2024-04-22T06:38:24.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47d2b1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 0x7fc478100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 -- 172.21.15.105:0/3955182123 learned_addr learned my addr 172.21.15.105:0/3955182123 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd876620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.934 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47dab2640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc47810c510 0x7fc4781018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 -- 172.21.15.105:0/2839020966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 msgr2=0x5556dd876620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd876620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 -- 172.21.15.105:0/2839020966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 msgr2=0x5556dd8824d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd8824d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 -- 172.21.15.105:0/2839020966 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556dd79a320 con 0x5556dd876b60 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d26ef640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 0x7f99b80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd8824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:24.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 -- 172.21.15.105:0/3955182123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 msgr2=0x7fc478100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 0x7fc478100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 -- 172.21.15.105:0/3955182123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc47810c510 msgr2=0x7fc4781018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc47810c510 0x7fc4781018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99c2ffd640 1 -- 172.21.15.105:0/2801101274 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f99c80d0d40 con 0x7f99cc10c5d0 2024-04-22T06:38:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009f60 tx=0x7f99c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d60f7ae3cc65424 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 -- 172.21.15.105:0/3955182123 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4781a6c40 con 0x7fc478103af0 2024-04-22T06:38:24.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 0x7fc478100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99b7fff640 1 -- 172.21.15.105:0/2839020966 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4016070 con 0x5556dd876b60 2024-04-22T06:38:24.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e9f0 tx=0x7fc46800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99d26ef640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 0x7f99b80a6430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99c80ce320 tx=0x7f99c806f600 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99b7fff640 1 -- 172.21.15.105:0/2839020966 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c401bb10 con 0x5556dd876b60 2024-04-22T06:38:24.938 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.931+0000 7f99b7fff640 1 -- 172.21.15.105:0/2839020966 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4012c30 con 0x5556dd876b60 2024-04-22T06:38:24.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc46e7fc640 1 -- 172.21.15.105:0/3955182123 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46800cd80 con 0x7fc478103af0 2024-04-22T06:38:24.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc46e7fc640 1 -- 172.21.15.105:0/3955182123 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680108e0 con 0x7fc478103af0 2024-04-22T06:38:24.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4781a6f30 con 0x7fc478103af0 2024-04-22T06:38:24.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc46e7fc640 1 -- 172.21.15.105:0/3955182123 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468018980 con 0x7fc478103af0 2024-04-22T06:38:24.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.931+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4781a7440 con 0x7fc478103af0 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 -- 172.21.15.105:0/2839020966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 msgr2=0x5556dd876fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009f60 tx=0x7f99c4009f90 comp rx=0 tx=0).stop 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 -- 172.21.15.105:0/2839020966 shutdown_connections 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd8824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd876fe0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2839020966 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd876620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 -- 172.21.15.105:0/2839020966 >> 172.21.15.105:0/2839020966 conn(0x5556dd83dc90 msgr2=0x5556dd871850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 -- 172.21.15.105:0/2839020966 shutdown_connections 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.932+0000 7f99d6e28740 1 -- 172.21.15.105:0/2839020966 wait complete. 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 -- start start 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 Processor -- start 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd87d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5556dd882e10 con 0x5556dd876b60 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5556dd763eb0 con 0x5556dd8800e0 2024-04-22T06:38:24.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.933+0000 7f99d6e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5556dd882f90 con 0x5556dd8778d0 2024-04-22T06:38:24.941 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.934+0000 7fc46e7fc640 1 -- 172.21.15.105:0/3955182123 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fc468018b20 con 0x7fc478103af0 2024-04-22T06:38:24.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d6e28740 10 client.0 _ll_register_callbacks cb 0x5556dd790b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33354/0 (socket says 172.21.15.105:33354) 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 -- 172.21.15.105:0/2527225881 learned_addr learned my addr 172.21.15.105:0/2527225881 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 -- 172.21.15.105:0/2527225881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 msgr2=0x5556dd87ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 -- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 msgr2=0x5556dd87d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd87d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.943 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 -- 172.21.15.105:0/2527225881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556dd4ee210 con 0x5556dd876b60 2024-04-22T06:38:24.943 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.934+0000 7fc46e7fc640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 0x7fc4640a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d16ed640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d647e80abc8a2078 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:24.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.934+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 0x7fc4640a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.944 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.934+0000 7fc46e7fc640 1 -- 172.21.15.105:0/3955182123 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc4680d0cc0 con 0x7fc478103af0 2024-04-22T06:38:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc046070 con 0x5556dd876b60 2024-04-22T06:38:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99c401bb10 con 0x5556dd876b60 2024-04-22T06:38:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.934+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f99bc02cd80 con 0x5556dd876b60 2024-04-22T06:38:24.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 Processor -- start 2024-04-22T06:38:24.945 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 -- start start 2024-04-22T06:38:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f99bc052050 con 0x5556dd876b60 2024-04-22T06:38:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 10 client.16050 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556dd5ba620 con 0x5556dd876b60 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfbe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 0x7fc4640a6410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc474006010 tx=0x7fc474005e40 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55722bf23500 con 0x55722bfbec30 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55722bec96a0 con 0x55722bfbe310 2024-04-22T06:38:24.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc4821e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55722bf1f4c0 con 0x55722bfc7810 2024-04-22T06:38:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c401bb10 con 0x5556dd876b60 2024-04-22T06:38:24.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33366/0 (socket says 172.21.15.105:33366) 2024-04-22T06:38:24.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 -- 172.21.15.105:0/3633348625 learned_addr learned my addr 172.21.15.105:0/3633348625 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.947 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfbe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f99bc0346b0 con 0x5556dd876b60 2024-04-22T06:38:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 10 client.16050 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 20 client.16050 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:24.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47dab2640 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd890130 0x5556dd892520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 4 client.16050 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556dd7ef880 con 0x5556dd890130 2024-04-22T06:38:24.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 -- 172.21.15.105:0/3633348625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 msgr2=0x55722bfc9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 -- 172.21.15.105:0/3633348625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 msgr2=0x55722bfbe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfbe6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 -- 172.21.15.105:0/3633348625 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722bee85e0 con 0x55722bfbec30 2024-04-22T06:38:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.935+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd890130 0x5556dd892520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.935+0000 7fc47cab0640 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc46800c6a0 tx=0x7fc4680a0e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86538b9d149a07b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4637fe640 1 -- 172.21.15.105:0/3633348625 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468010a90 con 0x55722bfbec30 2024-04-22T06:38:24.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4637fe640 1 -- 172.21.15.105:0/3633348625 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680991b0 con 0x55722bfbec30 2024-04-22T06:38:24.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4637fe640 1 -- 172.21.15.105:0/3633348625 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46809c1f0 con 0x55722bfbec30 2024-04-22T06:38:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.936+0000 7f99b57fa640 10 client.16050 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.936+0000 7f99d1eee640 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd890130 0x5556dd892520 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edfe5efc762670bf server_cookie=9e49b2c8f7ccf702 in_seq=0 out_seq=0 2024-04-22T06:38:24.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 -- 172.21.15.105:0/3633348625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 msgr2=0x55722bfc72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc46800c6a0 tx=0x7fc4680a0e40 comp rx=0 tx=0).stop 2024-04-22T06:38:24.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 -- 172.21.15.105:0/3633348625 shutdown_connections 2024-04-22T06:38:24.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfc72d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 --2- 172.21.15.105:0/3633348625 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfbe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 -- 172.21.15.105:0/3633348625 >> 172.21.15.105:0/3633348625 conn(0x55722bfb8a40 msgr2=0x55722bfb8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.936+0000 7fc4821e1740 1 -- 172.21.15.105:0/3633348625 shutdown_connections 2024-04-22T06:38:24.952 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 -- 172.21.15.105:0/3633348625 wait complete. 2024-04-22T06:38:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99b57fa640 1 -- 172.21.15.105:0/2527225881 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556dd7ef880 con 0x5556dd890130 2024-04-22T06:38:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99b57fa640 10 client.16050 handle_command_reply: tid=0 2024-04-22T06:38:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 client.16050 shutdown 2024-04-22T06:38:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 msgr2=0x5556dd87fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009900 tx=0x7f99bc009880 comp rx=0 tx=0).stop 2024-04-22T06:38:24.953 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 -- start start 2024-04-22T06:38:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 shutdown_connections 2024-04-22T06:38:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd890130 0x5556dd892520 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd87d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99b57fa640 0 client.16050 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 --2- 172.21.15.105:0/2527225881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd87fad0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 Processor -- start 2024-04-22T06:38:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 >> 172.21.15.105:0/2527225881 conn(0x5556dd83dc90 msgr2=0x5556dd8735a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 shutdown_connections 2024-04-22T06:38:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 1 -- 172.21.15.105:0/2527225881 wait complete. 2024-04-22T06:38:24.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.937+0000 7f99d6e28740 20 client.16050 trim_cache size 0 max 16384 2024-04-22T06:38:24.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbec30 0x55722bfc4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55722bfca5a0 con 0x55722bfbe310 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55722bedca60 con 0x55722bfbec30 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55722bfca720 con 0x55722bfc7810 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc4821e1740 10 client.0 _ll_register_callbacks cb 0x55722bc35110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc47d2b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc47d2b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33380/0 (socket says 172.21.15.105:33380) 2024-04-22T06:38:24.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.937+0000 7fc47d2b1640 1 -- 172.21.15.105:0/2899791853 learned_addr learned my addr 172.21.15.105:0/2899791853 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47cab0640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbec30 0x55722bfc4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 -- 172.21.15.105:0/2899791853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 msgr2=0x55722bfc5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 -- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbec30 msgr2=0x55722bfc4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbec30 0x55722bfc4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 -- 172.21.15.105:0/2899791853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722bc35920 con 0x55722bfbe310 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc474009900 tx=0x7fc474009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2672016073a97221 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474050070 con 0x55722bfbe310 2024-04-22T06:38:24.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc474003800 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4680991b0 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc47405c050 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc4821e1740 10 client.16056 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.938+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55722bd01d30 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680991b0 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc47403fbc0 con 0x55722bfbe310 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 10 client.16056 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 10 client.16056 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 20 client.16056 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722bfd7d30 0x55722bfda120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 4 client.16056 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55722bef6a10 con 0x55722bfd7d30 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc47cab0640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722bfd7d30 0x55722bfda120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc460ff9640 10 client.16056 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.939+0000 7fc47cab0640 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722bfd7d30 0x55722bfda120 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3902b7c8df1030c server_cookie=2863de8ec7c1ed6b in_seq=0 out_seq=0 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc460ff9640 1 -- 172.21.15.105:0/2899791853 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55722bef6a10 con 0x55722bfd7d30 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc460ff9640 10 client.16056 handle_command_reply: tid=0 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 client.16056 shutdown 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 msgr2=0x55722bfc4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc474009900 tx=0x7fc474009260 comp rx=0 tx=0).stop 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 shutdown_connections 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722bfd7d30 0x55722bfda120 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbec30 0x55722bfc4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 --2- 172.21.15.105:0/2899791853 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbe310 0x55722bfc4680 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 >> 172.21.15.105:0/2899791853 conn(0x55722bfb8a40 msgr2=0x55722bfba790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.940+0000 7fc460ff9640 0 client.16056 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.941+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 shutdown_connections 2024-04-22T06:38:24.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.941+0000 7fc4821e1740 1 -- 172.21.15.105:0/2899791853 wait complete. 2024-04-22T06:38:24.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.941+0000 7fc4821e1740 20 client.16056 trim_cache size 0 max 16384 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.952+0000 7f99d6e28740 1 Processor -- start 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.952+0000 7f99d6e28740 1 -- start start 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.952+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd876b60 0x5556dd88bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d6e28740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ad9b0 con 0x5556dd8800e0 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d6e28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1af4f0 con 0x5556dd876b60 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d6e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ae080 con 0x5556dd8778d0 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33396/0 (socket says 172.21.15.105:33396) 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 -- 172.21.15.105:0/1061194492 learned_addr learned my addr 172.21.15.105:0/1061194492 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d1eee640 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd876b60 0x5556dd88bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 -- 172.21.15.105:0/1061194492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 msgr2=0x5556dd87f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 -- 172.21.15.105:0/1061194492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd876b60 msgr2=0x5556dd88bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd876b60 0x5556dd88bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 -- 172.21.15.105:0/1061194492 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556dd875e30 con 0x5556dd8800e0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.953+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0096c0 tx=0x7f99bc02c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431620e1b73a3118 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.954+0000 7f99b77fe640 1 -- 172.21.15.105:0/1061194492 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc046040 con 0x5556dd8800e0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.954+0000 7f99b77fe640 1 -- 172.21.15.105:0/1061194492 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99bc034ca0 con 0x5556dd8800e0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.954+0000 7f99b77fe640 1 -- 172.21.15.105:0/1061194492 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc003cb0 con 0x5556dd8800e0 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- 172.21.15.105:0/1061194492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 msgr2=0x5556dd87fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0096c0 tx=0x7f99bc02c3d0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- 172.21.15.105:0/1061194492 shutdown_connections 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd8800e0 0x5556dd87fad0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd87f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1061194492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd876b60 0x5556dd88bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- 172.21.15.105:0/1061194492 >> 172.21.15.105:0/1061194492 conn(0x5556dd83dc90 msgr2=0x5556dd87b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- 172.21.15.105:0/1061194492 shutdown_connections 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- 172.21.15.105:0/1061194492 wait complete. 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- start start 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 Processor -- start 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd8813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd881920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1aee20 con 0x5556dd876b60 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ae750 con 0x5556dd8800e0 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ab990 con 0x5556dd8778d0 2024-04-22T06:38:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d6e28740 10 client.0 _ll_register_callbacks cb 0x7f99bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.955+0000 7f99d16ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33398/0 (socket says 172.21.15.105:33398) 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 -- 172.21.15.105:0/1424311856 learned_addr learned my addr 172.21.15.105:0/1424311856 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d1eee640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd8813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 -- 172.21.15.105:0/1424311856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 msgr2=0x5556dd8813e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd8813e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 -- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 msgr2=0x5556dd881920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd881920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 -- 172.21.15.105:0/1424311856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556dd8866c0 con 0x5556dd876b60 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d16ed640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f99bc02c4c0 tx=0x7f99bc02c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3750dba05dc359ef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc05c020 con 0x5556dd876b60 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99bc034ca0 con 0x5556dd876b60 2024-04-22T06:38:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7f99bc034850 con 0x5556dd876b60 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f99bc067050 con 0x5556dd876b60 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d6e28740 10 client.16062 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556dd8f6ac0 con 0x5556dd876b60 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.956+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99bc034ca0 con 0x5556dd876b60 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f99bc0046c0 con 0x5556dd876b60 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 10 client.16062 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 10 client.16062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 20 client.16062 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd8f7240 0x5556dd8f9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 4 client.16062 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5556dd77b3d0 con 0x5556dd8f7240 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d1eee640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd8f7240 0x5556dd8f9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99b57fa640 10 client.16062 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.957+0000 7f99d1eee640 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd8f7240 0x5556dd8f9660 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42b62aff80aa7762 server_cookie=dd496ddb8a74876c in_seq=0 out_seq=0 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 Processor -- start 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 -- start start 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfd3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781afbe0 con 0x55722bfbec30 2024-04-22T06:38:24.975 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781b1720 con 0x55722bfbe310 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc4821e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781b02b0 con 0x55722bfc7810 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33402/0 (socket says 172.21.15.105:33402) 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 -- 172.21.15.105:0/156635064 learned_addr learned my addr 172.21.15.105:0/156635064 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfd3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47dab2640 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 -- 172.21.15.105:0/156635064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 msgr2=0x55722bfc4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 -- 172.21.15.105:0/156635064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 msgr2=0x55722bfd3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfd3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 -- 172.21.15.105:0/156635064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722bfbea20 con 0x55722bfbec30 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc47cab0640 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc4680ce330 tx=0x7fc4680bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc58cbe38203bc5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc462ffd640 1 -- 172.21.15.105:0/156635064 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468010a90 con 0x55722bfbec30 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc462ffd640 1 -- 172.21.15.105:0/156635064 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680cebd0 con 0x55722bfbec30 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.959+0000 7fc462ffd640 1 -- 172.21.15.105:0/156635064 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680872a0 con 0x55722bfbec30 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 -- 172.21.15.105:0/156635064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 msgr2=0x55722bfb78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc4680ce330 tx=0x7fc4680bd040 comp rx=0 tx=0).stop 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 -- 172.21.15.105:0/156635064 shutdown_connections 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfc7810 0x55722bfc4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfb78e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 --2- 172.21.15.105:0/156635064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfbe310 0x55722bfd3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 -- 172.21.15.105:0/156635064 >> 172.21.15.105:0/156635064 conn(0x55722bfb8a40 msgr2=0x55722bfc4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 -- 172.21.15.105:0/156635064 shutdown_connections 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.960+0000 7fc4821e1740 1 -- 172.21.15.105:0/156635064 wait complete. 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 -- start start 2024-04-22T06:38:24.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 Processor -- start 2024-04-22T06:38:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99b57fa640 1 -- 172.21.15.105:0/1424311856 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5556dd77b3d0 con 0x5556dd8f7240 2024-04-22T06:38:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99b57fa640 10 client.16062 handle_command_reply: tid=0 2024-04-22T06:38:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 client.16062 shutdown 2024-04-22T06:38:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 msgr2=0x5556dd880ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f99bc02c4c0 tx=0x7f99bc02c760 comp rx=0 tx=0).stop 2024-04-22T06:38:24.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfbe310 0x55722bfce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfc7810 0x55722bfc18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781b1050 con 0x55722bfbec30 2024-04-22T06:38:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781b0980 con 0x55722bfc7810 2024-04-22T06:38:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 shutdown_connections 2024-04-22T06:38:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5556dd8f7240 0x5556dd8f9660 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fc4781adbc0 con 0x55722bfbe310 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5556dd8800e0 0x5556dd881920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99b57fa640 0 client.16062 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5556dd8778d0 0x5556dd8813e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 --2- 172.21.15.105:0/1424311856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5556dd876b60 0x5556dd880ea0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 >> 172.21.15.105:0/1424311856 conn(0x5556dd83dc90 msgr2=0x5556dd870040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.980 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc4821e1740 10 client.0 _ll_register_callbacks cb 0x55722bfc7760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 shutdown_connections 2024-04-22T06:38:24.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc47cab0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33418/0 (socket says 172.21.15.105:33418) 2024-04-22T06:38:24.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc47cab0640 1 -- 172.21.15.105:0/526464915 learned_addr learned my addr 172.21.15.105:0/526464915 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:24.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfbe310 0x55722bfce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.961+0000 7fc47dab2640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfc7810 0x55722bfc18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 1 -- 172.21.15.105:0/1424311856 wait complete. 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.961+0000 7f99d6e28740 20 client.16062 trim_cache size 0 max 16384 2024-04-22T06:38:24.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 -- 172.21.15.105:0/526464915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfbe310 msgr2=0x55722bfce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfbe310 0x55722bfce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:38:24.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:38:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 -- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfc7810 msgr2=0x55722bfc18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfc7810 0x55722bfc18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 -- 172.21.15.105:0/526464915 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722bfc1e20 con 0x55722bfbec30 2024-04-22T06:38:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc47cab0640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc46809cad0 tx=0x7fc4680a5780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf94cfbc877b0ca server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:24.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680ced80 con 0x55722bfbec30 2024-04-22T06:38:24.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8411+0+0 (secure 0 0 0) 0x7fc46809ae40 con 0x55722bfbec30 2024-04-22T06:38:24.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4680cebd0 con 0x55722bfbec30 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 msgr2=0x7f99b80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 0x7f99b80a6430 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99c80ce320 tx=0x7f99c806f600 comp rx=0 tx=0).stop 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 msgr2=0x7f99cc19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e970 tx=0x7f99c800ee30 comp rx=0 tx=0).stop 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 shutdown_connections 2024-04-22T06:38:24.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f99b80a3f70 0x7f99b80a6430 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc468010a90 con 0x55722bfbec30 2024-04-22T06:38:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99cc10c5d0 0x7f99cc19cf20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99cc103b20 0x7f99cc198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 --2- 172.21.15.105:0/2801101274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f99cc103200 0x7f99cc1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 >> 172.21.15.105:0/2801101274 conn(0x7f99cc0771d0 msgr2=0x7f99cc0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc4821e1740 10 client.16068 fetch_fsmap learned FSMap version 27 2024-04-22T06:38:24.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55722c03e790 con 0x55722bfbec30 2024-04-22T06:38:24.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 shutdown_connections 2024-04-22T06:38:24.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:24.962+0000 7f99d4152640 1 -- 172.21.15.105:0/2801101274 wait complete. 2024-04-22T06:38:24.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680cebd0 con 0x55722bfbec30 2024-04-22T06:38:24.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.962+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc468098890 con 0x55722bfbec30 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 10 client.16068 fetch_fsmap finished waiting for FSMap version 27 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 10 client.16068 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 20 client.16068 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722c03ea10 0x55722c040e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 4 client.16068 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2ad732f1", "release": true, "await": true}] 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2ad732f1", "release": true, "await": true}) -- 0x55722bf30f20 con 0x55722c03ea10 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722c03ea10 0x55722c040e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc47d2b1640 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722c03ea10 0x55722c040e00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db2223d1d03641f8 server_cookie=fcb2bd358280e74e in_seq=0 out_seq=0 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.963+0000 7fc460ff9640 10 client.16068 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.965+0000 7fc460ff9640 1 -- 172.21.15.105:0/526464915 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55722bf30f20 con 0x55722c03ea10 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.965+0000 7fc460ff9640 10 client.16068 handle_command_reply: tid=0 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.965+0000 7fc4821e1740 1 client.16068 shutdown 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 msgr2=0x55722bfce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc46809cad0 tx=0x7fc4680a5780 comp rx=0 tx=0).stop 2024-04-22T06:38:24.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 shutdown_connections 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55722c03ea10 0x55722c040e00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55722bfc7810 0x55722bfc18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55722bfbec30 0x55722bfce7a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 --2- 172.21.15.105:0/526464915 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55722bfbe310 0x55722bfce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc460ff9640 0 client.16068 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 >> 172.21.15.105:0/526464915 conn(0x55722bfb8a40 msgr2=0x55722bfc4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 shutdown_connections 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 1 -- 172.21.15.105:0/526464915 wait complete. 2024-04-22T06:38:24.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.966+0000 7fc4821e1740 20 client.16068 trim_cache size 0 max 16384 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 27, 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 4, 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "2ad732f1": { 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 4, 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:38:24.990 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 76.6, 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:24.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:24.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:24.992 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 msgr2=0x7fc4640a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 0x7fc4640a6410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc474006010 tx=0x7fc474005e40 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 msgr2=0x7fc478101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e9f0 tx=0x7fc46800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 shutdown_connections 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fc4640a3f50 0x7fc4640a6410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc47810c510 0x7fc4781018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478103af0 0x7fc478101360 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 --2- 172.21.15.105:0/3955182123 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc4781031d0 0x7fc478100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:24.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 >> 172.21.15.105:0/3955182123 conn(0x7fc478077f50 msgr2=0x7fc4780fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:24.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 shutdown_connections 2024-04-22T06:38:24.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:24.967+0000 7fc47f515640 1 -- 172.21.15.105:0/3955182123 wait complete. 2024-04-22T06:38:24.994 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 2ad732f1, seconds elapsed: 17.6 2024-04-22T06:38:25.111 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7834acb700 time 2024-04-22T06:38:24.964894+0000 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f78471043dd] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: clone() 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:24.965+0000 7f7834acb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7834acb700 time 2024-04-22T06:38:24.964894+0000 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f78471043dd] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.113 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: clone() 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: *** Caught signal (Aborted) ** 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: in thread 7f7834acb700 thread_name:quiesce.agt 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7845e8cd20] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: gsignal() 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: abort() 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7847104437] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.114 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 13: clone() 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:24.966+0000 7f7834acb700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.115 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: in thread 7f7834acb700 thread_name:quiesce.agt 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7845e8cd20] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: gsignal() 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: abort() 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7847104437] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.116 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 13: clone() 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: -1> 2024-04-22T06:38:24.965+0000 7f7834acb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7834acb700 time 2024-04-22T06:38:24.964894+0000 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f78471043dd] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.117 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: clone() 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 0> 2024-04-22T06:38:24.966+0000 7f7834acb700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: in thread 7f7834acb700 thread_name:quiesce.agt 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7845e8cd20] 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: gsignal() 2024-04-22T06:38:25.118 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: abort() 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7847104437] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 13: clone() 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: -1> 2024-04-22T06:38:24.965+0000 7f7834acb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7834acb700 time 2024-04-22T06:38:24.964894+0000 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.119 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f78471043dd] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: clone() 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.120 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 0> 2024-04-22T06:38:24.966+0000 7f7834acb700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: in thread 7f7834acb700 thread_name:quiesce.agt 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7845e8cd20] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2: gsignal() 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 3: abort() 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7847104437] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f78471045a3] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5589f72d7e9c] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5589f72d7f9f] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5589f72060d3] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5589f720603c] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5589f7485d3f] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5589f74ee560] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7845e821ca] 2024-04-22T06:38:25.121 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 13: clone() 2024-04-22T06:38:25.122 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.122 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[70213]: 2024-04-22T06:38:25.122 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:25 smithi105 podman[85770]: 2024-04-22 06:38:25.026228984 +0000 UTC m=+0.020429013 container died 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:38:25.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2086374609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:25.240 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5c4907f700 time 2024-04-22T06:38:24.964945+0000 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c5b6b83dd] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.241 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: clone() 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:24.965+0000 7f5c4907f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5c4907f700 time 2024-04-22T06:38:24.964945+0000 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c5b6b83dd] 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.242 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f097112e700 time 2024-04-22T06:38:24.965059+0000 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09837673dd] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.243 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: clone() 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: *** Caught signal (Aborted) ** 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: in thread 7f097112e700 thread_name:quiesce.agt 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:24.965+0000 7f097112e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f097112e700 time 2024-04-22T06:38:24.965059+0000 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.244 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: clone() 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: *** Caught signal (Aborted) ** 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: in thread 7f5c4907f700 thread_name:quiesce.agt 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c5a440d20] 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: gsignal() 2024-04-22T06:38:25.245 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: abort() 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c5b6b8437] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 13: clone() 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:24.966+0000 7f5c4907f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.246 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: in thread 7f5c4907f700 thread_name:quiesce.agt 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09837673dd] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.247 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: clone() 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f09824efd20] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: gsignal() 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: abort() 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0983767437] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.248 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 13: clone() 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:24.966+0000 7f097112e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: in thread 7f097112e700 thread_name:quiesce.agt 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.249 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c5a440d20] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: gsignal() 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: abort() 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c5b6b8437] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.250 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 13: clone() 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: -1> 2024-04-22T06:38:24.965+0000 7f5c4907f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5c4907f700 time 2024-04-22T06:38:24.964945+0000 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c5b6b83dd] 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.251 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f09824efd20] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: gsignal() 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: abort() 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0983767437] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.252 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 13: clone() 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: -2> 2024-04-22T06:38:24.965+0000 7f097112e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f097112e700 time 2024-04-22T06:38:24.965059+0000 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09837673dd] 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.253 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: clone() 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 0> 2024-04-22T06:38:24.966+0000 7f5c4907f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: in thread 7f5c4907f700 thread_name:quiesce.agt 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.254 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c5a440d20] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: gsignal() 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: abort() 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c5b6b8437] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.255 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 13: clone() 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.256 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: clone() 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 0> 2024-04-22T06:38:24.966+0000 7f097112e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: in thread 7f097112e700 thread_name:quiesce.agt 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f09824efd20] 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: gsignal() 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: abort() 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0983767437] 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.257 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 13: clone() 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.258 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: -1> 2024-04-22T06:38:24.965+0000 7f5c4907f700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f5c4907f700 time 2024-04-22T06:38:24.964945+0000 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f5c5b6b83dd] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.259 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: clone() 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 0> 2024-04-22T06:38:24.966+0000 7f5c4907f700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: in thread 7f5c4907f700 thread_name:quiesce.agt 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5c5a440d20] 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2: gsignal() 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 3: abort() 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f5c5b6b8437] 2024-04-22T06:38:25.260 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f5c5b6b85a3] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: -2> 2024-04-22T06:38:24.965+0000 7f097112e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f097112e700 time 2024-04-22T06:38:24.965059+0000 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f09837673dd] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.261 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: clone() 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 0> 2024-04-22T06:38:24.966+0000 7f097112e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: in thread 7f097112e700 thread_name:quiesce.agt 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f09824efd20] 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2: gsignal() 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 3: abort() 2024-04-22T06:38:25.262 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0983767437] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f09837675a3] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55d9e77a4e9c] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55d9e77a4f9f] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55d9e76d30d3] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55d9e76d303c] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55d9e7952d3f] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55d9e79bb560] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5c5a4361ca] 2024-04-22T06:38:25.263 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 13: clone() 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[61599]: 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:25 smithi177 podman[64927]: 2024-04-22 06:38:25.024988374 +0000 UTC m=+0.025596234 container died 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562af7c7de9c] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562af7c7df9f] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562af7bac0d3] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562af7bac03c] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562af7e2bd3f] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562af7e94560] 2024-04-22T06:38:25.264 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09824e51ca] 2024-04-22T06:38:25.265 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 13: clone() 2024-04-22T06:38:25.265 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.265 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[61909]: 2024-04-22T06:38:25.265 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:25 smithi177 podman[64928]: 2024-04-22 06:38:25.025476834 +0000 UTC m=+0.024060272 container died 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=) 2024-04-22T06:38:25.386 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2086374609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2086374609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1ad2697700 time 2024-04-22T06:38:24.964977+0000 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae4cd03dd] 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: clone() 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:24.965+0000 7f1ad2697700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1ad2697700 time 2024-04-22T06:38:24.964977+0000 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae4cd03dd] 2024-04-22T06:38:25.460 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.461 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.461 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.461 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.461 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.461 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: clone() 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: *** Caught signal (Aborted) ** 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: in thread 7f1ad2697700 thread_name:quiesce.agt 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1ae3a58d20] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: gsignal() 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: abort() 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae4cd0437] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.462 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 13: clone() 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:24.966+0000 7f1ad2697700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: in thread 7f1ad2697700 thread_name:quiesce.agt 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1ae3a58d20] 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: gsignal() 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: abort() 2024-04-22T06:38:25.463 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae4cd0437] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 13: clone() 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: -1> 2024-04-22T06:38:24.965+0000 7f1ad2697700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1ad2697700 time 2024-04-22T06:38:24.964977+0000 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.464 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae4cd03dd] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: clone() 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.465 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 0> 2024-04-22T06:38:24.966+0000 7f1ad2697700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: in thread 7f1ad2697700 thread_name:quiesce.agt 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1ae3a58d20] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: gsignal() 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: abort() 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae4cd0437] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.466 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 13: clone() 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: -1> 2024-04-22T06:38:24.965+0000 7f1ad2697700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f1ad2697700 time 2024-04-22T06:38:24.964977+0000 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1ae4cd03dd] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.467 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: clone() 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 0> 2024-04-22T06:38:24.966+0000 7f1ad2697700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: in thread 7f1ad2697700 thread_name:quiesce.agt 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1ae3a58d20] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2: gsignal() 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 3: abort() 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1ae4cd0437] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f1ae4cd05a3] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55c439890e9c] 2024-04-22T06:38:25.468 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55c439890f9f] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55c4397bf0d3] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55c4397bf03c] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55c439a3ed3f] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55c439aa7560] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f1ae3a4e1ca] 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 13: clone() 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[60731]: 2024-04-22T06:38:25.469 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:25 smithi112 podman[63929]: 2024-04-22 06:38:25.02558125 +0000 UTC m=+0.022170445 container died ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-22T06:38:26.080 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:25 smithi177 podman[64927]: 2024-04-22 06:38:25.770940802 +0000 UTC m=+0.771548707 container remove 26eafc33cbaa0d99de2359114249608d98e2b4be3cad9d66cb6d715f5c201ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:38:26.081 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:25 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:38:26.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:25 smithi105 podman[85770]: 2024-04-22 06:38:25.772077343 +0000 UTC m=+0.766277353 container remove 4dff5045448c62a24ee14e4ff9be14798c63a8b329151f7f3fd3cb0ad5afd3ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:38:26.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:25 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:38:26.112 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:26 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-22T06:38:26.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:26 smithi112 ceph-mon[31236]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T06:38:26.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:25 smithi112 podman[63929]: 2024-04-22 06:38:25.911152523 +0000 UTC m=+0.907741781 container remove ae424537b9fbfacbe18d9d28281b8ebbdb3754de2196ce17f7d610522b681e7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:38:26.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:25 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:38:26.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:26 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:38:26.355 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:26 smithi177 ceph-mon[31346]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T06:38:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:26 smithi105 ceph-mon[27916]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 853 B/s wr, 14 op/s 2024-04-22T06:38:26.740 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:26 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:38:26.740 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:26 smithi177 podman[64928]: 2024-04-22 06:38:26.358295732 +0000 UTC m=+1.356879170 container remove 33163aa97e7777540446057326d4be387c46edb9a714a816c2b67f4335008227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:38:26.740 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:26 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:38:26.740 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:26 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:38:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:28 smithi105 ceph-mon[27916]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T06:38:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:28 smithi112 ceph-mon[31236]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T06:38:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:28 smithi177 ceph-mon[31346]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-22T06:38:30.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:30 smithi105 ceph-mon[27916]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 341 B/s wr, 14 op/s 2024-04-22T06:38:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:30 smithi112 ceph-mon[31236]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 341 B/s wr, 14 op/s 2024-04-22T06:38:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:30 smithi177 ceph-mon[31346]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 341 B/s wr, 14 op/s 2024-04-22T06:38:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:32 smithi105 ceph-mon[27916]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T06:38:32.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:32 smithi112 ceph-mon[31236]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T06:38:32.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:32 smithi177 ceph-mon[31346]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-22T06:38:34.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:34 smithi105 ceph-mon[27916]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:38:34.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:34 smithi112 ceph-mon[31236]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:38:34.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:34 smithi177 ceph-mon[31346]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-22T06:38:36.398 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:36 smithi105 ceph-mon[27916]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T06:38:36.398 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:38:36.398 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 systemd[1]: Stopped Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:36.398 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 systemd[1]: Starting Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:38:36.442 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:36 smithi112 ceph-mon[31236]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T06:38:36.443 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:38:36.443 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 systemd[1]: Stopped Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:36.443 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 systemd[1]: Starting Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:38:36.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:36 smithi177 ceph-mon[31346]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-22T06:38:36.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 podman[85863]: 2024-04-22 06:38:36.303048266 +0000 UTC m=+0.021369471 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:38:36.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 podman[85863]: 2024-04-22 06:38:36.539569097 +0000 UTC m=+0.257890288 container create 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:38:36.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 podman[64035]: 2024-04-22 06:38:36.347743528 +0000 UTC m=+0.018086398 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:38:36.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 podman[64035]: 2024-04-22 06:38:36.445443883 +0000 UTC m=+0.115786767 container create d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:38:36.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 podman[64035]: 2024-04-22 06:38:36.689657497 +0000 UTC m=+0.360000368 container init d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:38:36.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 podman[64035]: 2024-04-22 06:38:36.692210311 +0000 UTC m=+0.362553192 container start d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:38:37.098 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: Stopped Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: Starting Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:36 smithi177 podman[65116]: 2024-04-22 06:38:36.894904638 +0000 UTC m=+0.020615526 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:36 smithi177 podman[65116]: 2024-04-22 06:38:36.991855761 +0000 UTC m=+0.117566630 container create dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux ) 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: Stopped Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:36 smithi177 systemd[1]: Starting Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:38:37.099 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:36 smithi177 podman[65117]: 2024-04-22 06:38:36.896276495 +0000 UTC m=+0.021190105 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:38:37.127 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 podman[85863]: 2024-04-22 06:38:36.792585993 +0000 UTC m=+0.510907186 container init 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 podman[85863]: 2024-04-22 06:38:36.795352704 +0000 UTC m=+0.513673895 container start 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 ceph-mds[85882]: -- 172.21.15.105:0/1693140384 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573f0bc3a40 con 0x5573f0b96800 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: starting mds.a at 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 bash[85863]: 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:38:36 smithi105 systemd[1]: Started Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 ceph-mds[64054]: -- 172.21.15.112:0/3069066330 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560f49d27a40 con 0x560f49cfb000 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: starting mds.h at 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 bash[64035]: d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e 2024-04-22T06:38:37.128 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:38:36 smithi112 systemd[1]: Started Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:37.394 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 podman[65117]: 2024-04-22 06:38:37.185833888 +0000 UTC m=+0.310747468 container create 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: daemon mds.h restarted 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:boot 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: Replacing daemon mds.c as rank 3 with standby daemon mds.l 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,3=c=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:38:37.395 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: daemon mds.a restarted 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:boot 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=k=up:replay,3=l=up:replay} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:38:37.396 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:replay 2 up:active 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: daemon mds.h restarted 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:boot 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: Replacing daemon mds.c as rank 3 with standby daemon mds.l 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,3=c=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:38:37.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: daemon mds.a restarted 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:boot 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=k=up:replay,3=l=up:replay} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:38:37.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:37 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:replay 2 up:active 2024-04-22T06:38:37.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: daemon mds.h restarted 2024-04-22T06:38:37.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:38:37.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: osdmap e93: 12 total, 12 up, 12 in 2024-04-22T06:38:37.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:boot 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: Replacing daemon mds.c as rank 3 with standby daemon mds.l 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,3=c=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: osdmap e94: 12 total, 12 up, 12 in 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: daemon mds.a restarted 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: osdmap e95: 12 total, 12 up, 12 in 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:boot 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=k=up:replay,3=l=up:replay} 3 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:38:37.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:37 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:replay 2 up:active 2024-04-22T06:38:37.653 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 podman[65117]: 2024-04-22 06:38:37.631075646 +0000 UTC m=+0.755989250 container init 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:38:37.653 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 podman[65117]: 2024-04-22 06:38:37.633639488 +0000 UTC m=+0.758553070 container start 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 podman[65116]: 2024-04-22 06:38:37.396940462 +0000 UTC m=+0.522651337 container init dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 podman[65116]: 2024-04-22 06:38:37.39955841 +0000 UTC m=+0.525269281 container start dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True) 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mds[65153]: -- 172.21.15.177:0/938295768 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563df27e9a40 con 0x563df27bd000 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: starting mds.i at 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 bash[65116]: dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b 2024-04-22T06:38:37.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:38:37 smithi177 systemd[1]: Started Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:37.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-mds[65174]: -- 172.21.15.177:0/3270081818 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d175c5a40 con 0x561d17598c00 2024-04-22T06:38:37.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: starting mds.c at 2024-04-22T06:38:37.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 bash[65117]: 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f 2024-04-22T06:38:37.990 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:38:37 smithi177 systemd[1]: Started Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.051+0000 7fe710c8c640 1 -- 172.21.15.105:0/3860701742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1021d0 msgr2=0x7fe70c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.051+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3860701742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1021d0 0x7fe70c1025b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe704009ca0 tx=0x7fe70402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 -- 172.21.15.105:0/3860701742 shutdown_connections 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3860701742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c1072a0 0x7fe70c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3860701742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c102b80 0x7fe70c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3860701742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1021d0 0x7fe70c1025b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 -- 172.21.15.105:0/3860701742 >> 172.21.15.105:0/3860701742 conn(0x7fe70c0fda50 msgr2=0x7fe70c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 -- 172.21.15.105:0/3860701742 shutdown_connections 2024-04-22T06:38:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 -- 172.21.15.105:0/3860701742 wait complete. 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 Processor -- start 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.052+0000 7fe710c8c640 1 -- start start 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 0x7fe70c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c074cd0 0x7fe70c1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c112200 con 0x7fe70c1072a0 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c112080 con 0x7fe70c074cd0 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c112380 con 0x7fe70c1021d0 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35668/0 (socket says 172.21.15.105:35668) 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 -- 172.21.15.105:0/3426465170 learned_addr learned my addr 172.21.15.105:0/3426465170 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 0x7fe70c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c074cd0 0x7fe70c1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 -- 172.21.15.105:0/3426465170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 msgr2=0x7fe70c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 0x7fe70c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 -- 172.21.15.105:0/3426465170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c074cd0 msgr2=0x7fe70c1a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c074cd0 0x7fe70c1a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 -- 172.21.15.105:0/3426465170 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70c1a6f30 con 0x7fe70c1072a0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 0x7fe70c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400d740 tx=0x7fe6f400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe7037fe640 1 -- 172.21.15.105:0/3426465170 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4017070 con 0x7fe70c1072a0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70c1a7220 con 0x7fe70c1072a0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe7037fe640 1 -- 172.21.15.105:0/3426465170 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6f400de50 con 0x7fe70c1072a0 2024-04-22T06:38:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.053+0000 7fe7037fe640 1 -- 172.21.15.105:0/3426465170 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4004ce0 con 0x7fe70c1072a0 2024-04-22T06:38:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.054+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70c1a7720 con 0x7fe70c1072a0 2024-04-22T06:38:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.056+0000 7fe7037fe640 1 -- 172.21.15.105:0/3426465170 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fe6f4004e80 con 0x7fe70c1072a0 2024-04-22T06:38:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.056+0000 7fe713882740 1 Processor -- start 2024-04-22T06:38:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.056+0000 7fe713882740 1 -- start start 2024-04-22T06:38:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.056+0000 7fe713882740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe713882740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe713882740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe713882740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555bb234a8d0 con 0x555bb23ef0e0 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe713882740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555bb22e7e80 con 0x555bb23e5b60 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe713882740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555bb2346890 con 0x555bb23e68d0 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35672/0 (socket says 172.21.15.105:35672) 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 -- 172.21.15.105:0/1558713460 learned_addr learned my addr 172.21.15.105:0/1558713460 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe70a59c640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe7037fe640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 0x7fe6f80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 -- 172.21.15.105:0/1558713460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 msgr2=0x555bb23e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23e5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 -- 172.21.15.105:0/1558713460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 msgr2=0x555bb23e5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 -- 172.21.15.105:0/1558713460 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb2309320 con 0x555bb23ef0e0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 0x7fe6f80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe7037fe640 1 -- 172.21.15.105:0/3426465170 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fe6f40d0580 con 0x7fe70c1072a0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.057+0000 7fe709d9b640 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40cdae0 tx=0x7fe6f4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76a6c5dba3adaa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe700ff9640 1 -- 172.21.15.105:0/1558713460 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4020070 con 0x555bb23ef0e0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe700ff9640 1 -- 172.21.15.105:0/1558713460 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6f409c8f0 con 0x555bb23ef0e0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe700ff9640 1 -- 172.21.15.105:0/1558713460 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4098860 con 0x555bb23ef0e0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 -- 172.21.15.105:0/1558713460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 msgr2=0x555bb23f14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40cdae0 tx=0x7fe6f4009c30 comp rx=0 tx=0).stop 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 0x7fe6f80a6430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00aba0 tx=0x7fe6fc0046c0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 -- 172.21.15.105:0/1558713460 shutdown_connections 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23ef0e0 0x555bb23f14d0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 --2- 172.21.15.105:0/1558713460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.058+0000 7fe713882740 1 -- 172.21.15.105:0/1558713460 >> 172.21.15.105:0/1558713460 conn(0x555bb23acc90 msgr2=0x555bb23e0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- 172.21.15.105:0/1558713460 shutdown_connections 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- 172.21.15.105:0/1558713460 wait complete. 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- start start 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 Processor -- start 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ebda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23ef0e0 0x555bb23ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555bb23f1db0 con 0x555bb23e5b60 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555bb22d2eb0 con 0x555bb23ef0e0 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x555bb23f1f30 con 0x555bb23e68d0 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe713882740 10 client.0 _ll_register_callbacks cb 0x555bb22ffb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35678/0 (socket says 172.21.15.105:35678) 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 -- 172.21.15.105:0/3525791037 learned_addr learned my addr 172.21.15.105:0/3525791037 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ebda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23ef0e0 0x555bb23ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 -- 172.21.15.105:0/3525791037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 msgr2=0x555bb23ebda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ebda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 -- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23ef0e0 msgr2=0x555bb23ec2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23ef0e0 0x555bb23ec2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe70a59c640 1 -- 172.21.15.105:0/3525791037 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb205d210 con 0x555bb23e5b60 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.059+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ebda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe704009ca0 tx=0x7fe70402ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1043f5507a2d6d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704044070 con 0x555bb23e5b60 2024-04-22T06:38:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fe7040326c0 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6f409c8f0 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe704050050 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 10 client.16104 fetch_fsmap learned FSMap version 33 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555bb2129620 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6f409c8f0 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 7765+0+0 (secure 0 0 0) 0x7fe7040043a0 con 0x555bb23e5b60 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 10 client.16104 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 10 client.16104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 20 client.16104 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23fefa0 0x555bb24013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 4 client.16104 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.060+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bb235e880 con 0x555bb23fefa0 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.061+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23fefa0 0x555bb24013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.061+0000 7fe6f1ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.061+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23fefa0 0x555bb24013b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5792523eb06363a3 server_cookie=a8cab8a20716ff28 in_seq=0 out_seq=0 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3525791037 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555bb235e880 con 0x555bb23fefa0 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe6f1ffb640 10 client.16104 handle_command_reply: tid=0 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 client.16104 shutdown 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 msgr2=0x555bb23eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe704009ca0 tx=0x7fe70402ce00 comp rx=0 tx=0).stop 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 shutdown_connections 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23fefa0 0x555bb24013b0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23ef0e0 0x555bb23ec2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe6f1ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ebda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 --2- 172.21.15.105:0/3525791037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e5b60 0x555bb23eea10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 >> 172.21.15.105:0/3525791037 conn(0x555bb23acc90 msgr2=0x555bb23e25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 shutdown_connections 2024-04-22T06:38:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 1 -- 172.21.15.105:0/3525791037 wait complete. 2024-04-22T06:38:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.062+0000 7fe713882740 20 client.16104 trim_cache size 0 max 16384 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.077+0000 7fe713882740 1 Processor -- start 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.077+0000 7fe713882740 1 -- start start 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b19a0 con 0x555bb23acc90 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1b2070 con 0x555bb23e5b60 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe713882740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1af8e0 con 0x555bb23e68d0 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35686/0 (socket says 172.21.15.105:35686) 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 -- 172.21.15.105:0/1035782980 learned_addr learned my addr 172.21.15.105:0/1035782980 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 -- 172.21.15.105:0/1035782980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 msgr2=0x555bb23ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 -- 172.21.15.105:0/1035782980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 msgr2=0x555bb23e50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe709d9b640 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 -- 172.21.15.105:0/1035782980 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bb23e4e30 con 0x555bb23acc90 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe70a59c640 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe704009a60 tx=0x7fe70402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73745deaebb4259a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe6f3fff640 1 -- 172.21.15.105:0/1035782980 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704044040 con 0x555bb23acc90 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe6f3fff640 1 -- 172.21.15.105:0/1035782980 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe704032970 con 0x555bb23acc90 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.078+0000 7fe6f3fff640 1 -- 172.21.15.105:0/1035782980 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704039040 con 0x555bb23acc90 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- 172.21.15.105:0/1035782980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 msgr2=0x555bb23facb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe704009a60 tx=0x7fe70402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- 172.21.15.105:0/1035782980 shutdown_connections 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e68d0 0x555bb23ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23e5b60 0x555bb23e50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 --2- 172.21.15.105:0/1035782980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23acc90 0x555bb23facb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- 172.21.15.105:0/1035782980 >> 172.21.15.105:0/1035782980 conn(0x555bb23ec440 msgr2=0x555bb23ea690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- 172.21.15.105:0/1035782980 shutdown_connections 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- 172.21.15.105:0/1035782980 wait complete. 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 -- start start 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.079+0000 7fe713882740 1 Processor -- start 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23acc90 0x555bb23f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e5b60 0x555bb23f7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1af210 con 0x555bb23e68d0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1ade40 con 0x555bb23acc90 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe70c1ae490 con 0x555bb23e5b60 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 10 client.0 _ll_register_callbacks cb 0x7fe704009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35702/0 (socket says 172.21.15.105:35702) 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 -- 172.21.15.105:0/3754997889 learned_addr learned my addr 172.21.15.105:0/3754997889 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe709d9b640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e5b60 0x555bb23f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23acc90 0x555bb23f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 -- 172.21.15.105:0/3754997889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e5b60 msgr2=0x555bb23f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e5b60 0x555bb23f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 -- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23acc90 msgr2=0x555bb23f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23acc90 0x555bb23f71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 -- 172.21.15.105:0/3754997889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bb23dbf30 con 0x555bb23e68d0 2024-04-22T06:38:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe70ad9d640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc01ce40 tx=0x7fe6fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1d179fc182d78a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe6f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc02b060 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fe6fc019df0 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.080+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe704032970 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe6fc032050 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 10 client.16110 fetch_fsmap learned FSMap version 33 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555bb2465c20 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe704032970 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 33) ==== 7765+0+0 (secure 0 0 0) 0x7fe6fc025030 con 0x555bb23e68d0 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 10 client.16110 fetch_fsmap finished waiting for FSMap version 33 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 20 client.16110 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23ef0e0 0x555bb2468410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 4 client.16110 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:38:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555bb22ea3d0 con 0x555bb23ef0e0 2024-04-22T06:38:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.081+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23ef0e0 0x555bb2468410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.082+0000 7fe6f1ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.082+0000 7fe70a59c640 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23ef0e0 0x555bb2468410 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e34a3102aa9fb80 server_cookie=861a4cf1b8fc2768 in_seq=0 out_seq=0 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe6f1ffb640 1 -- 172.21.15.105:0/3754997889 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555bb22ea3d0 con 0x555bb23ef0e0 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe6f1ffb640 10 client.16110 handle_command_reply: tid=0 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 client.16110 shutdown 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 msgr2=0x555bb23f01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc01ce40 tx=0x7fe6fc004730 comp rx=0 tx=0).stop 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 shutdown_connections 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x555bb23ef0e0 0x555bb2468410 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555bb23e68d0 0x555bb23f01e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x555bb23e5b60 0x555bb23f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 --2- 172.21.15.105:0/3754997889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555bb23acc90 0x555bb23f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 >> 172.21.15.105:0/3754997889 conn(0x555bb23ec440 msgr2=0x555bb23df040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe6f1ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 shutdown_connections 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.085+0000 7fe713882740 1 -- 172.21.15.105:0/3754997889 wait complete. 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe713882740 20 client.16110 trim_cache size 0 max 16384 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:38:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 msgr2=0x7fe6f80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 0x7fe6f80a6430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00aba0 tx=0x7fe6fc0046c0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 msgr2=0x7fe70c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400d740 tx=0x7fe6f400dc00 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 shutdown_connections 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe6f80a3f70 0x7fe6f80a6430 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe70c074cd0 0x7fe70c1a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe70c1072a0 0x7fe70c074790 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 --2- 172.21.15.105:0/3426465170 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe70c1021d0 0x7fe70c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 >> 172.21.15.105:0/3426465170 conn(0x7fe70c0fda50 msgr2=0x7fe70c0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 shutdown_connections 2024-04-22T06:38:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:38.086+0000 7fe710c8c640 1 -- 172.21.15.105:0/3426465170 wait complete. 2024-04-22T06:38:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:38:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: daemon mds.i restarted 2024-04-22T06:38:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:38:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:resolve 2024-04-22T06:38:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:resolve 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:resolve 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:boot 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:resolve,3=l=up:resolve,4=g=up:resolve} 2 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:38:38.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:38 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-04-22T06:38:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:38:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: daemon mds.i restarted 2024-04-22T06:38:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:38:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:resolve 2024-04-22T06:38:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:resolve 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:resolve 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:boot 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:resolve,3=l=up:resolve,4=g=up:resolve} 2 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:38:38.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:38 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: daemon mds.i restarted 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: osdmap e96: 12 total, 12 up, 12 in 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:resolve 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:resolve 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:resolve 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:boot 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:resolve,3=l=up:resolve,4=g=up:resolve} 2 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:38:38.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:38 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-04-22T06:38:39.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:39 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:boot 2024-04-22T06:38:39.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:39 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:resolve 2024-04-22T06:38:39.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:39 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:38:39.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:39 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:38:39.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:39 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:boot 2024-04-22T06:38:39.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:39 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:resolve 2024-04-22T06:38:39.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:39 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:38:39.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:39 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:38:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:39 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:boot 2024-04-22T06:38:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:39 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:resolve 2024-04-22T06:38:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:39 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:38:39.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:39 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:38:40.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 op/s 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.24973 172.21.15.105:0/183949266 after 0 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.24971 172.21.15.105:0/3177140182 after 0 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15432 172.21.15.105:0/975939081 after 0 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.000999974 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00199995 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.00199995 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:reconnect 2024-04-22T06:38:40.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:reconnect 2024-04-22T06:38:40.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:reconnect 2024-04-22T06:38:40.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:reconnect 2024-04-22T06:38:40.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:40 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 op/s 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.24973 172.21.15.105:0/183949266 after 0 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.24971 172.21.15.105:0/3177140182 after 0 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15432 172.21.15.105:0/975939081 after 0 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.000999974 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00199995 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.00199995 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:reconnect 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:reconnect 2024-04-22T06:38:40.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:reconnect 2024-04-22T06:38:40.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:reconnect 2024-04-22T06:38:40.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:40 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 16 op/s 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.24973 172.21.15.105:0/183949266 after 0 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.24971 172.21.15.105:0/3177140182 after 0 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15432 172.21.15.105:0/975939081 after 0 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999974 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.000999974 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00199995 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.15432 172.21.15.105:0/975939081 after 0.00199995 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: reconnect by client.24973 172.21.15.105:0/183949266 after 0.00199995 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:reconnect 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:reconnect 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:reconnect 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:reconnect 2024-04-22T06:38:40.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:40 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:38:41.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:rejoin 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:rejoin 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:rejoin 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:rejoin 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:active 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:active 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:active 2024-04-22T06:38:41.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:active 2024-04-22T06:38:41.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:41 smithi112 ceph-mon[31236]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:rejoin 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:rejoin 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:rejoin 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:rejoin 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:38:41.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:active 2024-04-22T06:38:41.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:active 2024-04-22T06:38:41.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:active 2024-04-22T06:38:41.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:active 2024-04-22T06:38:41.741 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:41 smithi177 ceph-mon[31346]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:rejoin 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:rejoin 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:rejoin 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:rejoin 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-04-22T06:38:41.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] up:active 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] up:active 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] up:active 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] up:active 2024-04-22T06:38:41.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:41 smithi105 ceph-mon[27916]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:38:42.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-mon[31236]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 15 op/s 2024-04-22T06:38:42.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:42.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:42.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:42.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:42 smithi177 ceph-mon[31346]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 15 op/s 2024-04-22T06:38:42.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:42 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:42.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:42 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:42.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:42 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:42 smithi105 ceph-mon[27916]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 15 op/s 2024-04-22T06:38:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:42 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:38:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:42 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:42.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:42 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:43.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:38:42.949+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:38:43.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:38:42.949+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:38:43.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:38:42.949+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-22T06:38:43.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:38:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:38:42.949+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-22T06:38:43.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:43.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:43.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:43.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:38:43.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:38:43.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:38:44.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:44 smithi177 ceph-mon[31346]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.4 KiB/s wr, 55 op/s 2024-04-22T06:38:44.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:44 smithi105 ceph-mon[27916]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.4 KiB/s wr, 55 op/s 2024-04-22T06:38:44.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:44 smithi112 ceph-mon[31236]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.4 KiB/s wr, 55 op/s 2024-04-22T06:38:46.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:38:46.725 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:38:46.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:46 smithi177 ceph-mon[31346]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 KiB/s wr, 48 op/s 2024-04-22T06:38:46.745 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:46 smithi105 ceph-mon[27916]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 KiB/s wr, 48 op/s 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 -- 172.21.15.105:0/1572854724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1874103ae0 msgr2=0x7f187410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 --2- 172.21.15.105:0/1572854724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1874103ae0 0x7f187410bfc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1868009900 tx=0x7f186802bd60 comp rx=0 tx=0).stop 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 -- 172.21.15.105:0/1572854724 shutdown_connections 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 --2- 172.21.15.105:0/1572854724 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f187410c500 0x7f187410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 --2- 172.21.15.105:0/1572854724 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1874103ae0 0x7f187410bfc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 --2- 172.21.15.105:0/1572854724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18741031c0 0x7f18741035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 -- 172.21.15.105:0/1572854724 >> 172.21.15.105:0/1572854724 conn(0x7f1874078000 msgr2=0x7f1874078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 -- 172.21.15.105:0/1572854724 shutdown_connections 2024-04-22T06:38:46.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.846+0000 7f187a952640 1 -- 172.21.15.105:0/1572854724 wait complete. 2024-04-22T06:38:46.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 Processor -- start 2024-04-22T06:38:46.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 -- start start 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1874103ae0 0x7f1874198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 0x7f187419cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1874112190 con 0x7f18741031c0 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1874112010 con 0x7f187410c500 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f187a952640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1874112310 con 0x7f1874103ae0 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f1873fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f1873fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57300/0 (socket says 172.21.15.105:57300) 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.847+0000 7f1873fff640 1 -- 172.21.15.105:0/487874136 learned_addr learned my addr 172.21.15.105:0/487874136 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f18737fe640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1874103ae0 0x7f1874198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1878eef640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 0x7f187419cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 -- 172.21.15.105:0/487874136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1874103ae0 msgr2=0x7f1874198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1874103ae0 0x7f1874198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 -- 172.21.15.105:0/487874136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 msgr2=0x7f187419cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 0x7f187419cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 -- 172.21.15.105:0/487874136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187419d6e0 con 0x7f18741031c0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1878eef640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 0x7f187419cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f1873fff640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f186400c890 tx=0x7f186400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f18717fa640 1 -- 172.21.15.105:0/487874136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864004220 con 0x7f18741031c0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f18717fa640 1 -- 172.21.15.105:0/487874136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18640043c0 con 0x7f18741031c0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f18717fa640 1 -- 172.21.15.105:0/487874136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864004a00 con 0x7f18741031c0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187419d9d0 con 0x7f18741031c0 2024-04-22T06:38:46.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.848+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18741a5260 con 0x7f18741031c0 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.851+0000 7f187d623740 1 Processor -- start 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.851+0000 7f187d623740 1 -- start start 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.851+0000 7f18717fa640 1 -- 172.21.15.105:0/487874136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f18640079b0 con 0x7f18741031c0 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.851+0000 7f187d623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.851+0000 7f187d623740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 0x556eccc64540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f187d623740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc64a80 0x556eccc66e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f187d623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556eccbc01e0 con 0x556eccc5b580 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f187d623740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556eccb5d390 con 0x556eccc64a80 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f187d623740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556eccbbc1a0 con 0x556eccc5bea0 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57302/0 (socket says 172.21.15.105:57302) 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 -- 172.21.15.105:0/1068139374 learned_addr learned my addr 172.21.15.105:0/1068139374 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f18737fe640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc64a80 0x556eccc66e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1873fff640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 0x556eccc64540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 -- 172.21.15.105:0/1068139374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 msgr2=0x556eccc64540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 0x556eccc64540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 -- 172.21.15.105:0/1068139374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc64a80 msgr2=0x556eccc66e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc64a80 0x556eccc66e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 -- 172.21.15.105:0/1068139374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ecc99c7d0 con 0x556eccc5b580 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1873fff640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 0x556eccc64540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f18717fa640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 0x7f185c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1873fff640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 0x7f185c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f1878eef640 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f186c009f30 tx=0x7f186c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146d9229f97814bf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.852+0000 7f18717fa640 1 -- 172.21.15.105:0/487874136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f18640d0920 con 0x7f18741031c0 2024-04-22T06:38:46.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.853+0000 7f185a7fc640 1 -- 172.21.15.105:0/1068139374 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c016070 con 0x556eccc5b580 2024-04-22T06:38:46.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.853+0000 7f185a7fc640 1 -- 172.21.15.105:0/1068139374 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c01bb10 con 0x556eccc5b580 2024-04-22T06:38:46.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.853+0000 7f185a7fc640 1 -- 172.21.15.105:0/1068139374 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c012c10 con 0x556eccc5b580 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- 172.21.15.105:0/1068139374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 msgr2=0x556eccc5b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f1873fff640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 0x7f185c0a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18640cdda0 tx=0x7f1864000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f186c009f30 tx=0x7f186c010800 comp rx=0 tx=0).stop 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- 172.21.15.105:0/1068139374 shutdown_connections 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc64a80 0x556eccc66e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5bea0 0x556eccc64540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 --2- 172.21.15.105:0/1068139374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5b580 0x556eccc5b960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- 172.21.15.105:0/1068139374 >> 172.21.15.105:0/1068139374 conn(0x556eccc55cb0 msgr2=0x556eccc56090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- 172.21.15.105:0/1068139374 shutdown_connections 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- 172.21.15.105:0/1068139374 wait complete. 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 -- start start 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.854+0000 7f187d623740 1 Processor -- start 2024-04-22T06:38:46.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 0x556eccc61ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc64a80 0x556eccc62560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556eccc6bc80 con 0x556eccc5bea0 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556eccb792f0 con 0x556eccc5b580 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x556eccc6be00 con 0x556eccc64a80 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f187d623740 10 client.0 _ll_register_callbacks cb 0x556eccb86580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57306/0 (socket says 172.21.15.105:57306) 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 -- 172.21.15.105:0/2710117039 learned_addr learned my addr 172.21.15.105:0/2710117039 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f1878eef640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc64a80 0x556eccc62560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f1878eef640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 0x556eccc61ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 -- 172.21.15.105:0/2710117039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc64a80 msgr2=0x556eccc62560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc64a80 0x556eccc62560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 -- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 msgr2=0x556eccc61ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 0x556eccc61ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 -- 172.21.15.105:0/2710117039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ecc8d23f0 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f1878eef640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 0x556eccc61ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18737fe640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1868005ec0 tx=0x7f186802cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334b8cd17ce6a0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.855+0000 7f18537fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868043070 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f1868005030 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f186c01bb10 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f186804f050 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 10 client.16122 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ecc8920f0 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c01bb10 con 0x556eccc5bea0 2024-04-22T06:38:46.859 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f18680035f0 con 0x556eccc5bea0 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 10 client.16122 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 20 client.16122 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccc78d70 0x556eccc7b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 4 client.16122 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.856+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556eccbcdc00 con 0x556eccc78d70 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.857+0000 7f1878eef640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccc78d70 0x556eccc7b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.857+0000 7f18537fe640 10 client.16122 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.857+0000 7f1878eef640 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccc78d70 0x556eccc7b1f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae671dc52e440ad server_cookie=3aff994f5956bf4c in_seq=0 out_seq=0 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f18537fe640 1 -- 172.21.15.105:0/2710117039 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556eccbcdc00 con 0x556eccc78d70 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f18537fe640 10 client.16122 handle_command_reply: tid=0 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 client.16122 shutdown 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 msgr2=0x556eccc62020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.860 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1868005ec0 tx=0x7f186802cb20 comp rx=0 tx=0).stop 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 shutdown_connections 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccc78d70 0x556eccc7b1f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc64a80 0x556eccc62560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc5bea0 0x556eccc62020 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 --2- 172.21.15.105:0/2710117039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5b580 0x556eccc61ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f18537fe640 0 client.16122 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 >> 172.21.15.105:0/2710117039 conn(0x556eccc55cb0 msgr2=0x556eccc57a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 shutdown_connections 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 1 -- 172.21.15.105:0/2710117039 wait complete. 2024-04-22T06:38:46.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.858+0000 7f187d623740 20 client.16122 trim_cache size 0 max 16384 2024-04-22T06:38:46.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.875+0000 7f187d623740 1 Processor -- start 2024-04-22T06:38:46.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.875+0000 7f187d623740 1 -- start start 2024-04-22T06:38:46.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc64210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.875 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc60a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18741abaf0 con 0x556eccc64a80 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18741ad630 con 0x556eccc5bea0 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f187d623740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18741ac1c0 con 0x556eccc5b580 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57320/0 (socket says 172.21.15.105:57320) 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 -- 172.21.15.105:0/3579387561 learned_addr learned my addr 172.21.15.105:0/3579387561 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc64210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f18737fe640 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc60a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 -- 172.21.15.105:0/3579387561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 msgr2=0x556eccc64210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc64210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 -- 172.21.15.105:0/3579387561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 msgr2=0x556eccc60a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc60a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 -- 172.21.15.105:0/3579387561 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eccc5bcd0 con 0x556eccc64a80 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1878eef640 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f186c012720 tx=0x7f186c010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d4925c5e59590a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.876 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.876+0000 7f1859ffb640 1 -- 172.21.15.105:0/3579387561 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c031070 con 0x556eccc64a80 2024-04-22T06:38:46.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.877+0000 7f1859ffb640 1 -- 172.21.15.105:0/3579387561 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c0298d0 con 0x556eccc64a80 2024-04-22T06:38:46.877 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.877+0000 7f1859ffb640 1 -- 172.21.15.105:0/3579387561 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c016040 con 0x556eccc64a80 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.877+0000 7f187d623740 1 -- 172.21.15.105:0/3579387561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 msgr2=0x556eccc74b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.877+0000 7f187d623740 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f186c012720 tx=0x7f186c010fd0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 -- 172.21.15.105:0/3579387561 shutdown_connections 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc74b40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc60a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 --2- 172.21.15.105:0/3579387561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc64210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 -- 172.21.15.105:0/3579387561 >> 172.21.15.105:0/3579387561 conn(0x556eccc55cb0 msgr2=0x556eccc62330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 -- 172.21.15.105:0/3579387561 shutdown_connections 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 -- 172.21.15.105:0/3579387561 wait complete. 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 -- start start 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 Processor -- start 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc71050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.878+0000 7f187d623740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc71590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f187d623740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f187d623740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18741acf60 con 0x556eccc64a80 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f187d623740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18741ac890 con 0x556eccc5bea0 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f187d623740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18741a9ad0 con 0x556eccc5b580 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f187d623740 10 client.0 _ll_register_callbacks cb 0x7f1868009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f18737fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc71590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57328/0 (socket says 172.21.15.105:57328) 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 -- 172.21.15.105:0/2101614703 learned_addr learned my addr 172.21.15.105:0/2101614703 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc71050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 -- 172.21.15.105:0/2101614703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 msgr2=0x556eccc71050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc71050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 -- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 msgr2=0x556eccc71590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc71590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.883 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 -- 172.21.15.105:0/2101614703 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eccc6ea00 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f18737fe640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc71590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f1878eef640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f186c010e00 tx=0x7f186c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b92a2d201b6f1d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f18537fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.879+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c016070 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f186c029620 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f186c0298d0 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f186c031040 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 10 client.16128 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ecccdffb0 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c0298d0 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f186c037040 con 0x556eccc64a80 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 10 client.16128 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 20 client.16128 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccce03d0 0x556eccce27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 4 client.16128 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556eccbc7a70 con 0x556eccce03d0 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.880+0000 7f18737fe640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccce03d0 0x556eccce27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f18537fe640 10 client.16128 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f18737fe640 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccce03d0 0x556eccce27c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b8b79265584eb server_cookie=f2cae74a8305c637 in_seq=0 out_seq=0 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f18537fe640 1 -- 172.21.15.105:0/2101614703 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556eccbc7a70 con 0x556eccce03d0 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f18537fe640 10 client.16128 handle_command_reply: tid=0 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f187d623740 1 client.16128 shutdown 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 msgr2=0x556eccc6e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f186c010e00 tx=0x7f186c010800 comp rx=0 tx=0).stop 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 shutdown_connections 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.881+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x556eccce03d0 0x556eccce27c0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f18537fe640 0 client.16128 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556eccc64a80 0x556eccc6e1d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556eccc5bea0 0x556eccc71590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 --2- 172.21.15.105:0/2101614703 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x556eccc5b580 0x556eccc71050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 >> 172.21.15.105:0/2101614703 conn(0x556eccc55cb0 msgr2=0x556eccc61a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 shutdown_connections 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 1 -- 172.21.15.105:0/2101614703 wait complete. 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187d623740 20 client.16128 trim_cache size 0 max 16384 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "no active scrubs running", 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": {} 2024-04-22T06:38:46.886 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 msgr2=0x7f185c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.882+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 0x7f185c0a6390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f18640cdda0 tx=0x7f1864000bc0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 msgr2=0x7f18741986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f186400c890 tx=0x7f186400cd50 comp rx=0 tx=0).stop 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 shutdown_connections 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f185c0a3ed0 0x7f185c0a6390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f187410c500 0x7f187419cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1874103ae0 0x7f1874198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.887 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 --2- 172.21.15.105:0/487874136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18741031c0 0x7f18741986f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:46.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 >> 172.21.15.105:0/487874136 conn(0x7f1874078000 msgr2=0x7f18740fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:46.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 shutdown_connections 2024-04-22T06:38:46.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:46.883+0000 7f187a952640 1 -- 172.21.15.105:0/487874136 wait complete. 2024-04-22T06:38:46.895 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:38:46.895 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:38:46.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:46 smithi112 ceph-mon[31236]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 1.4 KiB/s wr, 48 op/s 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 -- 172.21.15.105:0/3513878748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4102b40 msgr2=0x7fe2d4106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/3513878748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4102b40 0x7fe2d4106b80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0009900 tx=0x7fe2c002bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 -- 172.21.15.105:0/3513878748 shutdown_connections 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/3513878748 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4107260 0x7fe2d410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/3513878748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4102b40 0x7fe2d4106b80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/3513878748 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d4102190 0x7fe2d4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 -- 172.21.15.105:0/3513878748 >> 172.21.15.105:0/3513878748 conn(0x7fe2d40fd9f0 msgr2=0x7fe2d40ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 -- 172.21.15.105:0/3513878748 shutdown_connections 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.037+0000 7fe2dae5c640 1 -- 172.21.15.105:0/3513878748 wait complete. 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 Processor -- start 2024-04-22T06:38:47.036 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 -- start start 2024-04-22T06:38:47.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4102190 0x7fe2d4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.038 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 0x7fe2d419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d41121a0 con 0x7fe2d4107260 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d4112020 con 0x7fe2d419d000 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2dae5c640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d4112320 con 0x7fe2d4102190 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d8bf8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4102190 0x7fe2d4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57352/0 (socket says 172.21.15.105:57352) 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 -- 172.21.15.105:0/435381139 learned_addr learned my addr 172.21.15.105:0/435381139 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d93f9640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 0x7fe2d419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 -- 172.21.15.105:0/435381139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4102190 msgr2=0x7fe2d4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4102190 0x7fe2d4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 -- 172.21.15.105:0/435381139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 msgr2=0x7fe2d419d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 0x7fe2d419d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.039 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d3fff640 1 -- 172.21.15.105:0/435381139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d419daf0 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.038+0000 7fe2d93f9640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 0x7fe2d419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2d3fff640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0036040 tx=0x7fe2c00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0043020 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d419dd80 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2c002ca70 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0003ac0 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.039+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d41a55d0 con 0x7fe2d4107260 2024-04-22T06:38:47.040 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.040+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d4068de0 con 0x7fe2d4107260 2024-04-22T06:38:47.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.042+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fe2c0046020 con 0x7fe2d4107260 2024-04-22T06:38:47.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.043+0000 7fe2d1ffb640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 0x7fe2c40a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.043+0000 7fe2d8bf8640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 0x7fe2c40a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.041 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.043+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fe2c00f2c40 con 0x7fe2d4107260 2024-04-22T06:38:47.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.043+0000 7fe2d8bf8640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 0x7fe2c40a6410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009bd0 tx=0x7fe2c8005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.048+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2c004d020 con 0x7fe2d4107260 2024-04-22T06:38:47.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.325+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2d4102b40 con 0x7fe2d4107260 2024-04-22T06:38:47.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.326+0000 7fe2d1ffb640 1 -- 172.21.15.105:0/435381139 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11943 (secure 0 0 0) 0x7fe2c002cc20 con 0x7fe2d4107260 2024-04-22T06:38:47.325 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:38:47.325 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:38:41.031516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24431,"mds_2":24398,"mds_3":24397,"mds_4":14622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":31,"state":"up:active","state_seq":51,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":33,"state":"up:active","state_seq":55,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24397,14622,24398,24500,24431]},"id":1}]} 2024-04-22T06:38:47.325 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 37 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 msgr2=0x7fe2c40a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 0x7fe2c40a6410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009bd0 tx=0x7fe2c8005f90 comp rx=0 tx=0).stop 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 msgr2=0x7fe2d4198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe2c0036040 tx=0x7fe2c00045e0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 shutdown_connections 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe2c40a3f50 0x7fe2c40a6410 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2d419d000 0x7fe2d419d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4107260 0x7fe2d4198c80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 --2- 172.21.15.105:0/435381139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe2d4102190 0x7fe2d4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 >> 172.21.15.105:0/435381139 conn(0x7fe2d40fd9f0 msgr2=0x7fe2d40fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.328+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 shutdown_connections 2024-04-22T06:38:47.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.329+0000 7fe2dae5c640 1 -- 172.21.15.105:0/435381139 wait complete. 2024-04-22T06:38:47.338 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 -- 172.21.15.105:0/1616415887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 msgr2=0x7fb8f010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1616415887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f010dd30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009ca0 tx=0x7fb8e802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 -- 172.21.15.105:0/1616415887 shutdown_connections 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1616415887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f010dd30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1616415887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0101a40 0x7fb8f0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.457 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1616415887 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 0x7fb8f0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 -- 172.21.15.105:0/1616415887 >> 172.21.15.105:0/1616415887 conn(0x7fb8f0077f50 msgr2=0x7fb8f0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 -- 172.21.15.105:0/1616415887 shutdown_connections 2024-04-22T06:38:47.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.458+0000 7fb8f577e640 1 -- 172.21.15.105:0/1616415887 wait complete. 2024-04-22T06:38:47.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 Processor -- start 2024-04-22T06:38:47.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 -- start start 2024-04-22T06:38:47.459 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 0x7fb8f01994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 0x7fb8f01a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01110b0 con 0x7fb8f0106160 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f0110f30 con 0x7fb8f0199fc0 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8f577e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f0111230 con 0x7fb8f0101090 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57368/0 (socket says 172.21.15.105:57368) 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.459+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1736386919 learned_addr learned my addr 172.21.15.105:0/1736386919 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 0x7fb8f01994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 0x7fb8f01a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1736386919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 msgr2=0x7fb8f01994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 0x7fb8f01994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1736386919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 msgr2=0x7fb8f01a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 0x7fb8f01a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1736386919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f01a4d40 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 0x7fb8f01a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400c970 tx=0x7fb8e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8d7fff640 1 -- 172.21.15.105:0/1736386919 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4014070 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.460+0000 7fb8d7fff640 1 -- 172.21.15.105:0/1736386919 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8e4004440 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.461+0000 7fb8d7fff640 1 -- 172.21.15.105:0/1736386919 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4003e50 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.461+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f01a4fd0 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.461+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f01a5540 con 0x7fb8f0106160 2024-04-22T06:38:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.463+0000 7fb8d7fff640 1 -- 172.21.15.105:0/1736386919 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fb8e4017070 con 0x7fb8f0106160 2024-04-22T06:38:47.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 Processor -- start 2024-04-22T06:38:47.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 -- start start 2024-04-22T06:38:47.463 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8d7fff640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 0x7fb8d80a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 0x55efb8d83430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 0x7fb8d80a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8d7fff640 1 -- 172.21.15.105:0/1736386919 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb8e40d0940 con 0x7fb8f0106160 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 0x55efb8d85e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efb8cdf1f0 con 0x55efb8d7a4d0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55efb8c7c4b0 con 0x55efb8d83a00 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8f836a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55efb8cdb1b0 con 0x55efb8d7ae80 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.464+0000 7fb8eeffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8eeffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57372/0 (socket says 172.21.15.105:57372) 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8eeffd640 1 -- 172.21.15.105:0/1828713489 learned_addr learned my addr 172.21.15.105:0/1828713489 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 0x55efb8d83430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 0x55efb8d85e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 0x7fb8d80a65b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009430 tx=0x7fb8e802cd60 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.464 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 -- 172.21.15.105:0/1828713489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 msgr2=0x55efb8d83430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 0x55efb8d83430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 -- 172.21.15.105:0/1828713489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 msgr2=0x55efb8d7a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 -- 172.21.15.105:0/1828713489 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb8c6e1b0 con 0x55efb8d83a00 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 0x55efb8d83430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 0x55efb8d85e50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8e802c700 tx=0x7fb8e8045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a430e9b2fdbcdc5a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.465+0000 7fb8d57fa640 1 -- 172.21.15.105:0/1828713489 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8047040 con 0x55efb8d83a00 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.466+0000 7fb8d57fa640 1 -- 172.21.15.105:0/1828713489 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8e8003e20 con 0x55efb8d83a00 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.466+0000 7fb8d57fa640 1 -- 172.21.15.105:0/1828713489 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8039040 con 0x55efb8d83a00 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.466+0000 7fb8f836a740 1 -- 172.21.15.105:0/1828713489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 msgr2=0x55efb8d85e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.466+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 0x55efb8d85e50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8e802c700 tx=0x7fb8e8045000 comp rx=0 tx=0).stop 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 -- 172.21.15.105:0/1828713489 shutdown_connections 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d83a00 0x55efb8d85e50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d7ae80 0x55efb8d83430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1828713489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d7a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 -- 172.21.15.105:0/1828713489 >> 172.21.15.105:0/1828713489 conn(0x55efb8d74ba0 msgr2=0x55efb8d74f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 -- 172.21.15.105:0/1828713489 shutdown_connections 2024-04-22T06:38:47.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 -- 172.21.15.105:0/1828713489 wait complete. 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 -- start start 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 Processor -- start 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d83260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.467+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d80b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8f836a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55efb8d86910 con 0x55efb8d7ae80 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8f836a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55efb8c98500 con 0x55efb8d7a4d0 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8f836a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55efb8d86a90 con 0x55efb8d83a00 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8f836a740 10 client.0 _ll_register_callbacks cb 0x55efb8c66b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57376/0 (socket says 172.21.15.105:57376) 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8eeffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d83260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1126605185 learned_addr learned my addr 172.21.15.105:0/1126605185 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d80b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1126605185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 msgr2=0x55efb8d80b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d80b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1126605185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 msgr2=0x55efb8d83260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d83260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/1126605185 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb89f13f0 con 0x55efb8d7ae80 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb8e401f000 tx=0x7fb8e409cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6626fdb1fb191c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.468+0000 7fb8d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.469+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40ba060 con 0x55efb8d7ae80 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.469+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb8e409bc00 con 0x55efb8d7ae80 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.469+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e8003e20 con 0x55efb8d7ae80 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.470+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb8e40bb250 con 0x55efb8d7ae80 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.470+0000 7fb8f836a740 10 client.16146 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.470+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55efb8abb7d0 con 0x55efb8d7ae80 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.472+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8e8003e20 con 0x55efb8d7ae80 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fb8e4098300 con 0x55efb8d7ae80 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 10 client.16146 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 10 client.16146 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.g 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 20 client.16146 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8d93bc0 0x55efb8d95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 4 client.16146 mds_command: new command op to 14622 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:47.471 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 --> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efb8c7e930 con 0x55efb8d93bc0 2024-04-22T06:38:47.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8d93bc0 0x55efb8d95fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8d93bc0 0x55efb8d95fb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9b5101cfd95a60fc server_cookie=32aead3d89254390 in_seq=0 out_seq=0 2024-04-22T06:38:47.472 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.473+0000 7fb8d2ff5640 10 client.16146 ms_handle_connect on v2:172.21.15.105:6838/256609249 2024-04-22T06:38:47.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/1126605185 <== mds.4 v2:172.21.15.105:6838/256609249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efb8c7e930 con 0x55efb8d93bc0 2024-04-22T06:38:47.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8d2ff5640 10 client.16146 handle_command_reply: tid=0 2024-04-22T06:38:47.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 client.16146 shutdown 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 msgr2=0x55efb8d805f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb8e401f000 tx=0x7fb8e409cdf0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 shutdown_connections 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8d93bc0 0x55efb8d95fb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d80b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d805f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 --2- 172.21.15.105:0/1126605185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d83260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 >> 172.21.15.105:0/1126605185 conn(0x55efb8d74ba0 msgr2=0x55efb8d768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8d2ff5640 0 client.16146 ms_handle_reset on v2:172.21.15.105:6838/256609249 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 shutdown_connections 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 1 -- 172.21.15.105:0/1126605185 wait complete. 2024-04-22T06:38:47.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.475+0000 7fb8f836a740 20 client.16146 trim_cache size 0 max 16384 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.500+0000 7fb8f836a740 1 Processor -- start 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.500+0000 7fb8f836a740 1 -- start start 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d82d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01adc10 con 0x55efb8d7ae80 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01af750 con 0x55efb8d7a4d0 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8f836a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01ae2e0 con 0x55efb8d83a00 2024-04-22T06:38:47.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57382/0 (socket says 172.21.15.105:57382) 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/568842833 learned_addr learned my addr 172.21.15.105:0/568842833 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/568842833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 msgr2=0x55efb8d82d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d82d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/568842833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 msgr2=0x55efb8d8f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 -- 172.21.15.105:0/568842833 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb8d70470 con 0x55efb8d7ae80 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.501+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4098f80 tx=0x7fb8e40038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146a8dc76d5a68c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.502+0000 7fb8d4ff9640 1 -- 172.21.15.105:0/568842833 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e409ba00 con 0x55efb8d7ae80 2024-04-22T06:38:47.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.502+0000 7fb8d4ff9640 1 -- 172.21.15.105:0/568842833 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8e409bba0 con 0x55efb8d7ae80 2024-04-22T06:38:47.502 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.502+0000 7fb8d4ff9640 1 -- 172.21.15.105:0/568842833 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e409b7f0 con 0x55efb8d7ae80 2024-04-22T06:38:47.505 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- 172.21.15.105:0/568842833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 msgr2=0x55efb8d73a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4098f80 tx=0x7fb8e40038b0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- 172.21.15.105:0/568842833 shutdown_connections 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d82d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7ae80 0x55efb8d73a80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 --2- 172.21.15.105:0/568842833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- 172.21.15.105:0/568842833 >> 172.21.15.105:0/568842833 conn(0x55efb8d74ba0 msgr2=0x55efb8d73370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- 172.21.15.105:0/568842833 shutdown_connections 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- 172.21.15.105:0/568842833 wait complete. 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 -- start start 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.503+0000 7fb8f836a740 1 Processor -- start 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7ae80 0x55efb8d7d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d7d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01af080 con 0x55efb8d7a4d0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01ae9b0 con 0x55efb8d7ae80 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01abc10 con 0x55efb8d83a00 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8f836a740 10 client.0 _ll_register_callbacks cb 0x7fb8e409bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57386/0 (socket says 172.21.15.105:57386) 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 -- 172.21.15.105:0/2806444555 learned_addr learned my addr 172.21.15.105:0/2806444555 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7ae80 0x55efb8d7d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d7d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 -- 172.21.15.105:0/2806444555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 msgr2=0x55efb8d7d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d7d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 -- 172.21.15.105:0/2806444555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7ae80 msgr2=0x55efb8d7d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7ae80 0x55efb8d7d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 -- 172.21.15.105:0/2806444555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb8df8f80 con 0x55efb8d7a4d0 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8ef7fe640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d7d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8eeffd640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000a640 tx=0x7fb8e0010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ce91e1fa5f847e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8d2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.504+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0005440 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb8e00055e0 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e409bba0 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb8e002f050 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 10 client.16152 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55efb8dfa110 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8e409bba0 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fb8e0020640 con 0x55efb8d7a4d0 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 10 client.16152 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 10 client.16152 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.g 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 20 client.16152 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8dfa960 0x55efb8dfcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 4 client.16152 mds_command: new command op to 14622 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:47.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 --> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55efb8d80ec0 con 0x55efb8dfa960 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.505+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8dfa960 0x55efb8dfcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8d2ff5640 10 client.16152 ms_handle_connect on v2:172.21.15.105:6838/256609249 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8ee7fc640 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8dfa960 0x55efb8dfcd80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8ef9c4c223e48447 server_cookie=701415faad10d0 in_seq=0 out_seq=0 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8d2ff5640 1 -- 172.21.15.105:0/2806444555 <== mds.4 v2:172.21.15.105:6838/256609249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55efb8d80ec0 con 0x55efb8dfa960 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8d2ff5640 10 client.16152 handle_command_reply: tid=0 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 client.16152 shutdown 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 msgr2=0x55efb8d8c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000a640 tx=0x7fb8e0010af0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 shutdown_connections 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x55efb8dfa960 0x55efb8dfcd80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55efb8d83a00 0x55efb8d7d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55efb8d7ae80 0x55efb8d7d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 --2- 172.21.15.105:0/2806444555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efb8d7a4d0 0x55efb8d8c250 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 >> 172.21.15.105:0/2806444555 conn(0x55efb8d74ba0 msgr2=0x55efb8d7baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8d2ff5640 0 client.16152 ms_handle_reset on v2:172.21.15.105:6838/256609249 2024-04-22T06:38:47.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 shutdown_connections 2024-04-22T06:38:47.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 1 -- 172.21.15.105:0/2806444555 wait complete. 2024-04-22T06:38:47.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.506+0000 7fb8f836a740 20 client.16152 trim_cache size 0 max 16384 2024-04-22T06:38:47.510 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:47.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 msgr2=0x7fb8d80a65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 0x7fb8d80a65b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009430 tx=0x7fb8e802cd60 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 msgr2=0x7fb8f0199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400c970 tx=0x7fb8e400ce30 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 shutdown_connections 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb8d80a40f0 0x7fb8d80a65b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb8f0199fc0 0x7fb8f01a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8f0106160 0x7fb8f0199a30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 --2- 172.21.15.105:0/1736386919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8f0101090 0x7fb8f01994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 >> 172.21.15.105:0/1736386919 conn(0x7fb8f0077f50 msgr2=0x7fb8f00fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 shutdown_connections 2024-04-22T06:38:47.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.507+0000 7fb8f577e640 1 -- 172.21.15.105:0/1736386919 wait complete. 2024-04-22T06:38:47.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 -- 172.21.15.105:0/488597055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 msgr2=0x7fdba4101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 --2- 172.21.15.105:0/488597055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4101470 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009900 tx=0x7fdba002bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 -- 172.21.15.105:0/488597055 shutdown_connections 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 --2- 172.21.15.105:0/488597055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba4106160 0x7fdba410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 --2- 172.21.15.105:0/488597055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4101a40 0x7fdba4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 --2- 172.21.15.105:0/488597055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4101470 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.661+0000 7fdbac21f640 1 -- 172.21.15.105:0/488597055 >> 172.21.15.105:0/488597055 conn(0x7fdba4077f50 msgr2=0x7fdba4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.662+0000 7fdbac21f640 1 -- 172.21.15.105:0/488597055 shutdown_connections 2024-04-22T06:38:47.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.662+0000 7fdbac21f640 1 -- 172.21.15.105:0/488597055 wait complete. 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.662+0000 7fdbac21f640 1 Processor -- start 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.662+0000 7fdbac21f640 1 -- start start 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4106160 0x7fdba4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 0x7fdba41a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41110c0 con 0x7fdba4101090 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdba4110f40 con 0x7fdba419a0e0 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbac21f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdba4111240 con 0x7fdba4106160 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57408/0 (socket says 172.21.15.105:57408) 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 -- 172.21.15.105:0/883075450 learned_addr learned my addr 172.21.15.105:0/883075450 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba97ba640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4106160 0x7fdba4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 0x7fdba41a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 -- 172.21.15.105:0/883075450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4106160 msgr2=0x7fdba4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4106160 0x7fdba4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 -- 172.21.15.105:0/883075450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 msgr2=0x7fdba41a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 0x7fdba41a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 -- 172.21.15.105:0/883075450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba41a4e60 con 0x7fdba4101090 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 0x7fdba41a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdba0005940 tx=0x7fdba002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.663+0000 7fdb9affd640 1 -- 172.21.15.105:0/883075450 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0043020 con 0x7fdba4101090 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.664+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba41a50f0 con 0x7fdba4101090 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.664+0000 7fdb9affd640 1 -- 172.21.15.105:0/883075450 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdba002ca70 con 0x7fdba4101090 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.664+0000 7fdb9affd640 1 -- 172.21.15.105:0/883075450 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00039f0 con 0x7fdba4101090 2024-04-22T06:38:47.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.664+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba41a5690 con 0x7fdba4101090 2024-04-22T06:38:47.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdb9affd640 1 -- 172.21.15.105:0/883075450 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fdba0046020 con 0x7fdba4101090 2024-04-22T06:38:47.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 Processor -- start 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 -- start start 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 0x558f061c3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f061c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558f0611f1f0 con 0x558f061ba4d0 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f060bc4b0 con 0x558f061bae80 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558f0611b1b0 con 0x558f061c3a00 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaa7bc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaa7bc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57410/0 (socket says 172.21.15.105:57410) 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/1868619797 learned_addr learned my addr 172.21.15.105:0/1868619797 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.667+0000 7fdb9affd640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 0x7fdb900a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdba97ba640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f061c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 0x558f061c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.667 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdb9affd640 1 -- 172.21.15.105:0/883075450 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fdba00f2c70 con 0x7fdba4101090 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/1868619797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 msgr2=0x558f061c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f061c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/1868619797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 msgr2=0x558f061c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 0x558f061c3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/1868619797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f060ae1b0 con 0x558f061ba4d0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 0x558f061c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 0x7fdb900a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c0128e0 tx=0x7fdb9c012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d1df23dddafa00 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdb8ffff640 1 -- 172.21.15.105:0/1868619797 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01fb10 con 0x558f061ba4d0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdb8ffff640 1 -- 172.21.15.105:0/1868619797 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c01fcb0 con 0x558f061ba4d0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdb8ffff640 1 -- 172.21.15.105:0/1868619797 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01ee10 con 0x558f061ba4d0 2024-04-22T06:38:47.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.668+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 0x7fdb900a6390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c025000 tx=0x7fdb9c022000 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 -- 172.21.15.105:0/1868619797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 msgr2=0x558f061ba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c0128e0 tx=0x7fdb9c012da0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 -- 172.21.15.105:0/1868619797 shutdown_connections 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f061c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061bae80 0x558f061c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/1868619797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061ba8b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.669+0000 7fdbaee0d740 1 -- 172.21.15.105:0/1868619797 >> 172.21.15.105:0/1868619797 conn(0x558f061b4ba0 msgr2=0x558f061b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.670+0000 7fdbaee0d740 1 -- 172.21.15.105:0/1868619797 shutdown_connections 2024-04-22T06:38:47.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/435381139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.670+0000 7fdbaee0d740 1 -- 172.21.15.105:0/1868619797 wait complete. 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.670+0000 7fdbaee0d740 1 -- start start 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.670+0000 7fdbaee0d740 1 Processor -- start 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061c3a00 0x558f061c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558f061c6970 con 0x558f061ba4d0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f060d8500 con 0x558f061c3a00 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558f061c6af0 con 0x558f061bae80 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57412/0 (socket says 172.21.15.105:57412) 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 -- 172.21.15.105:0/2183126001 learned_addr learned my addr 172.21.15.105:0/2183126001 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaee0d740 10 client.0 _ll_register_callbacks cb 0x558f060a6b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba97ba640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061c3a00 0x558f061c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 -- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 msgr2=0x558f061c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 -- 172.21.15.105:0/2183126001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061c3a00 msgr2=0x558f061c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061c3a00 0x558f061c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdba9fbb640 1 -- 172.21.15.105:0/2183126001 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f05e313f0 con 0x558f061ba4d0 2024-04-22T06:38:47.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.671+0000 7fdb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdba00bb3c0 tx=0x7fdba004fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6a6d59e2ba0e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00bee50 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb9c01fcb0 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fdba00bec50 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdba00bee50 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 10 client.16164 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558f05efb7d0 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c01fcb0 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fdba00c71d0 con 0x558f061ba4d0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 10 client.16164 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 10 client.16164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 20 client.16164 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.672+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f061d39e0 0x558f061d5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.673+0000 7fdbaee0d740 4 client.16164 mds_command: new command op to 24397 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.673+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 --> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f060be930 con 0x558f061d39e0 2024-04-22T06:38:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.673+0000 7fdba97ba640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f061d39e0 0x558f061d5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.673+0000 7fdb8d7fa640 10 client.16164 ms_handle_connect on v2:172.21.15.177:6832/624977252 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.673+0000 7fdba97ba640 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f061d39e0 0x558f061d5e50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a8c1a509df708d2f server_cookie=fa7abe3d36d84958 in_seq=0 out_seq=0 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/2183126001 <== mds.3 v2:172.21.15.177:6832/624977252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f060be930 con 0x558f061d39e0 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdb8d7fa640 10 client.16164 handle_command_reply: tid=0 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 client.16164 shutdown 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 msgr2=0x558f061c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdba00bb3c0 tx=0x7fdba004fd50 comp rx=0 tx=0).stop 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 shutdown_connections 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f061d39e0 0x558f061d5e50 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061c3a00 0x558f061c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdb8d7fa640 0 client.16164 ms_handle_reset on v2:172.21.15.177:6832/624977252 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/2183126001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061ba4d0 0x558f061c31e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.674+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 >> 172.21.15.105:0/2183126001 conn(0x558f061b4ba0 msgr2=0x558f061b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.675+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 shutdown_connections 2024-04-22T06:38:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.675+0000 7fdbaee0d740 1 -- 172.21.15.105:0/2183126001 wait complete. 2024-04-22T06:38:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.675+0000 7fdbaee0d740 20 client.16164 trim_cache size 0 max 16384 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 Processor -- start 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 -- start start 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41abae0 con 0x558f061c3a00 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41ad620 con 0x558f061ba4d0 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41ac1b0 con 0x558f061bae80 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57416/0 (socket says 172.21.15.105:57416) 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/3749669482 learned_addr learned my addr 172.21.15.105:0/3749669482 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdba97ba640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/3749669482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 msgr2=0x558f061c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/3749669482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 msgr2=0x558f061cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdbaa7bc640 1 -- 172.21.15.105:0/3749669482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f061c0e10 con 0x558f061c3a00 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.698+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009e80 tx=0x7fdb9c01e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9957d0c843c2c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdb8f7fe640 1 -- 172.21.15.105:0/3749669482 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c03a070 con 0x558f061c3a00 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdb8f7fe640 1 -- 172.21.15.105:0/3749669482 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c01ea60 con 0x558f061c3a00 2024-04-22T06:38:47.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdb8f7fe640 1 -- 172.21.15.105:0/3749669482 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01e850 con 0x558f061c3a00 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3749669482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 msgr2=0x558f061c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.699+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009e80 tx=0x7fdb9c01e9f0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3749669482 shutdown_connections 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061c3a00 0x558f061c31e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061bae80 0x558f061c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3749669482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3749669482 >> 172.21.15.105:0/3749669482 conn(0x558f061b4ba0 msgr2=0x558f061b6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3749669482 shutdown_connections 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3749669482 wait complete. 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 -- start start 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 Processor -- start 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.700+0000 7fdbaee0d740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f06238f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41acf50 con 0x558f061bae80 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41ac880 con 0x558f061ba4d0 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaee0d740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdba41a9b00 con 0x558f061c3a00 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba97ba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaee0d740 10 client.0 _ll_register_callbacks cb 0x7fdba00bac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:51252/0 (socket says 172.21.15.105:51252) 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba97ba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57426/0 (socket says 172.21.15.105:57426) 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 -- 172.21.15.105:0/3875515167 learned_addr learned my addr 172.21.15.105:0/3875515167 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f06238f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 -- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 msgr2=0x558f06238f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f06238f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 -- 172.21.15.105:0/3875515167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 msgr2=0x558f061bdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 -- 172.21.15.105:0/3875515167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f062394d0 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba97ba640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaa7bc640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f06238f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdba9fbb640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdba0006010 tx=0x7fdba0003dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4071f5077154b1b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00d9070 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb9c01ea60 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.701+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fdba00be050 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdba00be530 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 10 client.25484 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558f062394d0 con 0x558f061ba4d0 2024-04-22T06:38:47.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb9c01ea60 con 0x558f061ba4d0 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fdba00ba740 con 0x558f061ba4d0 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 10 client.25484 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 10 client.25484 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 20 client.25484 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f0623a9d0 0x558f0623cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 4 client.25484 mds_command: new command op to 24397 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 --> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f0612cc10 con 0x558f0623a9d0 2024-04-22T06:38:47.707 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.702+0000 7fdba97ba640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f0623a9d0 0x558f0623cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdb8d7fa640 10 client.25484 ms_handle_connect on v2:172.21.15.177:6832/624977252 2024-04-22T06:38:47.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdba97ba640 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f0623a9d0 0x558f0623cdc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae87d7e9f6c4ee8 server_cookie=88fb89aa502a2131 in_seq=0 out_seq=0 2024-04-22T06:38:47.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdb8d7fa640 1 -- 172.21.15.105:0/3875515167 <== mds.3 v2:172.21.15.177:6832/624977252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f0612cc10 con 0x558f0623a9d0 2024-04-22T06:38:47.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdb8d7fa640 10 client.25484 handle_command_reply: tid=0 2024-04-22T06:38:47.708 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 client.25484 shutdown 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 msgr2=0x558f061bd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdba0006010 tx=0x7fdba0003dc0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 shutdown_connections 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:6832/624977252,v1:172.21.15.177:6833/624977252] conn(0x558f0623a9d0 0x558f0623cdc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558f061c3a00 0x558f06238f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdb8d7fa640 0 client.25484 ms_handle_reset on v2:172.21.15.177:6832/624977252 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558f061bae80 0x558f061bdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.709 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 --2- 172.21.15.105:0/3875515167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f061ba4d0 0x558f061bd5b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 >> 172.21.15.105:0/3875515167 conn(0x558f061b4ba0 msgr2=0x558f061b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 shutdown_connections 2024-04-22T06:38:47.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.703+0000 7fdbaee0d740 1 -- 172.21.15.105:0/3875515167 wait complete. 2024-04-22T06:38:47.710 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbaee0d740 20 client.25484 trim_cache size 0 max 16384 2024-04-22T06:38:47.710 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:47.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 msgr2=0x7fdb900a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 0x7fdb900a6390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c025000 tx=0x7fdb9c022000 comp rx=0 tx=0).stop 2024-04-22T06:38:47.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 msgr2=0x7fdba4199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.711 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fdba0005940 tx=0x7fdba002c5f0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 shutdown_connections 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdb900a3ed0 0x7fdb900a6390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdba419a0e0 0x7fdba41a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdba4106160 0x7fdba4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 --2- 172.21.15.105:0/883075450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdba4101090 0x7fdba4199660 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 >> 172.21.15.105:0/883075450 conn(0x7fdba4077f50 msgr2=0x7fdba40ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 shutdown_connections 2024-04-22T06:38:47.712 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.704+0000 7fdbac21f640 1 -- 172.21.15.105:0/883075450 wait complete. 2024-04-22T06:38:47.716 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-22T06:38:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/435381139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:47.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 -- 172.21.15.105:0/3708264747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24101090 msgr2=0x7fca24101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3708264747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24101090 0x7fca24101470 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fca18009900 tx=0x7fca1802bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 -- 172.21.15.105:0/3708264747 shutdown_connections 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3708264747 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24106160 0x7fca2410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3708264747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca24101a40 0x7fca24105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3708264747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24101090 0x7fca24101470 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 -- 172.21.15.105:0/3708264747 >> 172.21.15.105:0/3708264747 conn(0x7fca24077f50 msgr2=0x7fca24078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 -- 172.21.15.105:0/3708264747 shutdown_connections 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 -- 172.21.15.105:0/3708264747 wait complete. 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.820+0000 7fca2ab96640 1 Processor -- start 2024-04-22T06:38:47.819 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 -- start start 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24101090 0x7fca241995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57454/0 (socket says 172.21.15.105:57454) 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 0x7fca241a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 -- 172.21.15.105:0/3071347729 learned_addr learned my addr 172.21.15.105:0/3071347729 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca241110a0 con 0x7fca24106160 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca24110f20 con 0x7fca2419a070 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca24111220 con 0x7fca24101090 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca28932640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24101090 0x7fca241995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca29133640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 0x7fca241a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 -- 172.21.15.105:0/3071347729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24101090 msgr2=0x7fca241995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24101090 0x7fca241995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 -- 172.21.15.105:0/3071347729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 msgr2=0x7fca241a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 0x7fca241a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca23fff640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca241a4e20 con 0x7fca24106160 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.821+0000 7fca29133640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 0x7fca241a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca23fff640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e9c0 tx=0x7fca1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca21ffb640 1 -- 172.21.15.105:0/3071347729 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1000cd80 con 0x7fca24106160 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca21ffb640 1 -- 172.21.15.105:0/3071347729 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca100054d0 con 0x7fca24106160 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca241a5110 con 0x7fca24106160 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca21ffb640 1 -- 172.21.15.105:0/3071347729 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10010430 con 0x7fca24106160 2024-04-22T06:38:47.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.822+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca241a5660 con 0x7fca24106160 2024-04-22T06:38:47.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.824+0000 7fca21ffb640 1 -- 172.21.15.105:0/3071347729 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fca10004fd0 con 0x7fca24106160 2024-04-22T06:38:47.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca21ffb640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 0x7fca140a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca21ffb640 1 -- 172.21.15.105:0/3071347729 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fca100d1060 con 0x7fca24106160 2024-04-22T06:38:47.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca28932640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 0x7fca140a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca2d788740 1 Processor -- start 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca2d788740 1 -- start start 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.825+0000 7fca28932640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 0x7fca140a6410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fca18006010 tx=0x7fca18005e40 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc7258b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc730e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559cfc68a1f0 con 0x559cfc725e80 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559cfc6274b0 con 0x559cfc72ea00 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca2d788740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cfc6861b0 con 0x559cfc7254d0 2024-04-22T06:38:47.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57468/0 (socket says 172.21.15.105:57468) 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 -- 172.21.15.105:0/748494065 learned_addr learned my addr 172.21.15.105:0/748494065 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca28932640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc7258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 -- 172.21.15.105:0/748494065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 msgr2=0x559cfc7258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc7258b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 -- 172.21.15.105:0/748494065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 msgr2=0x559cfc730e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca29133640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc730e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc730e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 -- 172.21.15.105:0/748494065 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cfc6191b0 con 0x559cfc725e80 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca29133640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc730e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.826+0000 7fca23fff640 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fca100ce5c0 tx=0x7fca1009c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733a99c2cba0f9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca0effd640 1 -- 172.21.15.105:0/748494065 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1009ccd0 con 0x559cfc725e80 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca0effd640 1 -- 172.21.15.105:0/748494065 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca100a1080 con 0x559cfc725e80 2024-04-22T06:38:47.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca0effd640 1 -- 172.21.15.105:0/748494065 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100ce790 con 0x559cfc725e80 2024-04-22T06:38:47.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 -- 172.21.15.105:0/748494065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 msgr2=0x559cfc72e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fca100ce5c0 tx=0x7fca1009c630 comp rx=0 tx=0).stop 2024-04-22T06:38:47.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 -- 172.21.15.105:0/748494065 shutdown_connections 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc730e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc72e430 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 --2- 172.21.15.105:0/748494065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc7258b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 -- 172.21.15.105:0/748494065 >> 172.21.15.105:0/748494065 conn(0x559cfc71fba0 msgr2=0x559cfc71ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 -- 172.21.15.105:0/748494065 shutdown_connections 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.827+0000 7fca2d788740 1 -- 172.21.15.105:0/748494065 wait complete. 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.828+0000 7fca2d788740 1 -- start start 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.828+0000 7fca2d788740 1 Processor -- start 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.828+0000 7fca2d788740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.828+0000 7fca2d788740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 0x559cfc72b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.828+0000 7fca2d788740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc72ea00 0x559cfc72bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca2d788740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559cfc7318b0 con 0x559cfc72ea00 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca2d788740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559cfc643500 con 0x559cfc725e80 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca2d788740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x559cfc731a30 con 0x559cfc7254d0 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca2d788740 10 client.0 _ll_register_callbacks cb 0x559cfc611b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca28932640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca28932640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53214/0 (socket says 172.21.15.105:53214) 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca28932640 1 -- 172.21.15.105:0/3986068713 learned_addr learned my addr 172.21.15.105:0/3986068713 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 0x559cfc72b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca29133640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc72ea00 0x559cfc72bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 -- 172.21.15.105:0/3986068713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 msgr2=0x559cfc72e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 -- 172.21.15.105:0/3986068713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc72ea00 msgr2=0x559cfc72bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc72ea00 0x559cfc72bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 -- 172.21.15.105:0/3986068713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cfc39c3f0 con 0x559cfc725e80 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca28932640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca23fff640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 0x559cfc72b720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x559cfc720c10 tx=0x7fca100ce4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=987192972eda4e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca03fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1009ccd0 con 0x559cfc725e80 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fca100bd0e0 con 0x559cfc725e80 2024-04-22T06:38:47.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.829+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca100a1080 con 0x559cfc725e80 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fca100bd300 con 0x559cfc725e80 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 10 client.25493 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559cfc4667d0 con 0x559cfc725e80 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca100a5b90 con 0x559cfc725e80 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fca100ce790 con 0x559cfc725e80 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 10 client.25493 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 10 client.25493 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.k 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 20 client.25493 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc73e9d0 0x559cfc740dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 4 client.25493 mds_command: new command op to 24398 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 --> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fca100a1080 con 0x559cfc73e9d0 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.830+0000 7fca28932640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc73e9d0 0x559cfc740dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.831+0000 7fca28932640 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc73e9d0 0x559cfc740dc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d3178d10120a17b server_cookie=39f9af7d86fab18b in_seq=0 out_seq=0 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.831+0000 7fca03fff640 10 client.25493 ms_handle_connect on v2:172.21.15.112:6832/583171810 2024-04-22T06:38:47.833 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca03fff640 1 -- 172.21.15.105:0/3986068713 <== mds.2 v2:172.21.15.112:6832/583171810 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fca100a1080 con 0x559cfc73e9d0 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca03fff640 10 client.25493 handle_command_reply: tid=0 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 client.25493 shutdown 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 msgr2=0x559cfc72b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 0x559cfc72b720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x559cfc720c10 tx=0x7fca100ce4a0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 shutdown_connections 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc73e9d0 0x559cfc740dc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc72ea00 0x559cfc72bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc725e80 0x559cfc72b720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 --2- 172.21.15.105:0/3986068713 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc7254d0 0x559cfc72e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 >> 172.21.15.105:0/3986068713 conn(0x559cfc71fba0 msgr2=0x559cfc7218f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca03fff640 0 client.25493 ms_handle_reset on v2:172.21.15.112:6832/583171810 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 shutdown_connections 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 1 -- 172.21.15.105:0/3986068713 wait complete. 2024-04-22T06:38:47.834 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.832+0000 7fca2d788740 20 client.25493 trim_cache size 0 max 16384 2024-04-22T06:38:47.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 Processor -- start 2024-04-22T06:38:47.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 -- start start 2024-04-22T06:38:47.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc725e80 0x559cfc73a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc72de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca241abb00 con 0x559cfc7254d0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca241ad640 con 0x559cfc72ea00 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca2d788740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca241ac1d0 con 0x559cfc725e80 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca28932640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca28932640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57476/0 (socket says 172.21.15.105:57476) 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca28932640 1 -- 172.21.15.105:0/4104035532 learned_addr learned my addr 172.21.15.105:0/4104035532 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca29133640 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc72de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.856+0000 7fca23fff640 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc725e80 0x559cfc73a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 -- 172.21.15.105:0/4104035532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc725e80 msgr2=0x559cfc73a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc725e80 0x559cfc73a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 -- 172.21.15.105:0/4104035532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 msgr2=0x559cfc72de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc72de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 -- 172.21.15.105:0/4104035532 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cfc725c20 con 0x559cfc7254d0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca28932640 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca18043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33f3cc62c63a849 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca0e7fc640 1 -- 172.21.15.105:0/4104035532 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18005280 con 0x559cfc7254d0 2024-04-22T06:38:47.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca0e7fc640 1 -- 172.21.15.105:0/4104035532 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca1805a070 con 0x559cfc7254d0 2024-04-22T06:38:47.857 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.857+0000 7fca0e7fc640 1 -- 172.21.15.105:0/4104035532 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18004a20 con 0x559cfc7254d0 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- 172.21.15.105:0/4104035532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 msgr2=0x559cfc72a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fca18000c00 tx=0x7fca18043000 comp rx=0 tx=0).stop 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- 172.21.15.105:0/4104035532 shutdown_connections 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc72ea00 0x559cfc72de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc725e80 0x559cfc73a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 --2- 172.21.15.105:0/4104035532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc7254d0 0x559cfc72a880 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- 172.21.15.105:0/4104035532 >> 172.21.15.105:0/4104035532 conn(0x559cfc71fba0 msgr2=0x559cfc739c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- 172.21.15.105:0/4104035532 shutdown_connections 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- 172.21.15.105:0/4104035532 wait complete. 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 -- start start 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.858+0000 7fca2d788740 1 Processor -- start 2024-04-22T06:38:47.861 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc741110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc72ea00 0x559cfc7a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca241acf70 con 0x559cfc725e80 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fca241ac8a0 con 0x559cfc7254d0 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fca241a9b40 con 0x559cfc72ea00 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca28932640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca2d788740 10 client.0 _ll_register_callbacks cb 0x7fca1000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca23fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc741110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca28932640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:51306/0 (socket says 172.21.15.105:51306) 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca28932640 1 -- 172.21.15.105:0/2983844214 learned_addr learned my addr 172.21.15.105:0/2983844214 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.859+0000 7fca29133640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc72ea00 0x559cfc7a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 -- 172.21.15.105:0/2983844214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc72ea00 msgr2=0x559cfc7a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc72ea00 0x559cfc7a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 -- 172.21.15.105:0/2983844214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 msgr2=0x559cfc741110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc741110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 -- 172.21.15.105:0/2983844214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cfc7a30e0 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca23fff640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc741110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca28932640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca1803a090 tx=0x7fca1802cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f26f06e4e4c69d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca03fff640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca18062070 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fca18055340 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca1805a070 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fca18055540 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca2d788740 10 client.25499 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559cfc7a4340 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.860+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca1805a070 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fca18003820 con 0x559cfc7254d0 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 10 client.25499 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 10 client.25499 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.k 2024-04-22T06:38:47.863 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 20 client.25499 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc7a4b20 0x559cfc7a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 4 client.25499 mds_command: new command op to 24398 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 --> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559cfc629930 con 0x559cfc7a4b20 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca23fff640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc7a4b20 0x559cfc7a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca03fff640 10 client.25499 ms_handle_connect on v2:172.21.15.112:6832/583171810 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.861+0000 7fca23fff640 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc7a4b20 0x559cfc7a6f40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a252c1f2c52ddb4a server_cookie=35cab51ac745e548 in_seq=0 out_seq=0 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca03fff640 1 -- 172.21.15.105:0/2983844214 <== mds.2 v2:172.21.15.112:6832/583171810 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559cfc629930 con 0x559cfc7a4b20 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca03fff640 10 client.25499 handle_command_reply: tid=0 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 client.25499 shutdown 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 msgr2=0x559cfc740bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fca1803a090 tx=0x7fca1802cdd0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 shutdown_connections 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:6832/583171810,v1:172.21.15.112:6833/583171810] conn(0x559cfc7a4b20 0x559cfc7a6f40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x559cfc72ea00 0x559cfc7a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca03fff640 0 client.25499 ms_handle_reset on v2:172.21.15.112:6832/583171810 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559cfc725e80 0x559cfc741110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 --2- 172.21.15.105:0/2983844214 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559cfc7254d0 0x559cfc740bd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 >> 172.21.15.105:0/2983844214 conn(0x559cfc71fba0 msgr2=0x559cfc73a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 shutdown_connections 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 1 -- 172.21.15.105:0/2983844214 wait complete. 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.862+0000 7fca2d788740 20 client.25499 trim_cache size 0 max 16384 2024-04-22T06:38:47.865 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 msgr2=0x7fca140a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 0x7fca140a6410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fca18006010 tx=0x7fca18005e40 comp rx=0 tx=0).stop 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 msgr2=0x7fca24199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e9c0 tx=0x7fca1000ee80 comp rx=0 tx=0).stop 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 shutdown_connections 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fca140a3f50 0x7fca140a6410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fca2419a070 0x7fca241a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca24106160 0x7fca24199ae0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 --2- 172.21.15.105:0/3071347729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fca24101090 0x7fca241995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 >> 172.21.15.105:0/3071347729 conn(0x7fca24077f50 msgr2=0x7fca240ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 shutdown_connections 2024-04-22T06:38:47.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.863+0000 7fca2ab96640 1 -- 172.21.15.105:0/3071347729 wait complete. 2024-04-22T06:38:47.875 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-22T06:38:47.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/435381139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 -- 172.21.15.105:0/1333541698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524102190 msgr2=0x7f5524102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 --2- 172.21.15.105:0/1333541698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524102190 0x7f5524102570 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5514009900 tx=0x7f551402bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 -- 172.21.15.105:0/1333541698 shutdown_connections 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 --2- 172.21.15.105:0/1333541698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5524107260 0x7f552410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 --2- 172.21.15.105:0/1333541698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102b40 0x7f5524106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 --2- 172.21.15.105:0/1333541698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524102190 0x7f5524102570 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 -- 172.21.15.105:0/1333541698 >> 172.21.15.105:0/1333541698 conn(0x7f55240793a0 msgr2=0x7f55240797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 -- 172.21.15.105:0/1333541698 shutdown_connections 2024-04-22T06:38:47.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.986+0000 7f5528bef640 1 -- 172.21.15.105:0/1333541698 wait complete. 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 Processor -- start 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 -- start start 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102190 0x7f552419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55241985c0 0x7f5524198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55241121a0 con 0x7f5524107260 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5524112020 con 0x7f5524102190 2024-04-22T06:38:47.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5528bef640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5524112320 con 0x7f55241985c0 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.987+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57504/0 (socket says 172.21.15.105:57504) 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 -- 172.21.15.105:0/238340047 learned_addr learned my addr 172.21.15.105:0/238340047 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f552259c640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102190 0x7f552419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 -- 172.21.15.105:0/238340047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55241985c0 msgr2=0x7f5524198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55241985c0 0x7f5524198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 -- 172.21.15.105:0/238340047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102190 msgr2=0x7f552419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102190 0x7f552419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 -- 172.21.15.105:0/238340047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55241990e0 con 0x7f5524107260 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5521d9b640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f551800e9c0 tx=0x7f551800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f55137fe640 1 -- 172.21.15.105:0/238340047 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551800cd80 con 0x7f5524107260 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55241993d0 con 0x7f5524107260 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55241a5670 con 0x7f5524107260 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f55137fe640 1 -- 172.21.15.105:0/238340047 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55180054d0 con 0x7f5524107260 2024-04-22T06:38:47.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.988+0000 7f55137fe640 1 -- 172.21.15.105:0/238340047 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5518010430 con 0x7f5524107260 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 Processor -- start 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 -- start start 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e4a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e55e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e428daf1f0 con 0x55e428e53a00 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e428d4c4b0 con 0x55e428e4ae80 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552b7dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e428dab1b0 con 0x55e428e4a4d0 2024-04-22T06:38:47.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552259c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:51344/0 (socket says 172.21.15.105:51344) 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f552259c640 1 -- 172.21.15.105:0/3103647658 learned_addr learned my addr 172.21.15.105:0/3103647658 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f5522d9d640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e4a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.991+0000 7f55137fe640 1 -- 172.21.15.105:0/238340047 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f5518005020 con 0x7f5524107260 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 -- 172.21.15.105:0/3103647658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 msgr2=0x55e428e4a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e4a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 -- 172.21.15.105:0/3103647658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 msgr2=0x55e428e55e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f5521d9b640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e55e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e55e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 -- 172.21.15.105:0/3103647658 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e428d3e1b0 con 0x55e428e4ae80 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f5521d9b640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f552259c640 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5514009900 tx=0x7f5514005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6646e215d9f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f55137fe640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 0x7f550c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f5510ff9640 1 -- 172.21.15.105:0/3103647658 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514044020 con 0x55e428e4ae80 2024-04-22T06:38:47.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f5522d9d640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 0x7f550c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.992+0000 7f5510ff9640 1 -- 172.21.15.105:0/3103647658 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551402cc90 con 0x55e428e4ae80 2024-04-22T06:38:47.993 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f55137fe640 1 -- 172.21.15.105:0/238340047 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f55180d10b0 con 0x7f5524107260 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f5510ff9640 1 -- 172.21.15.105:0/3103647658 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55140046c0 con 0x55e428e4ae80 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 -- 172.21.15.105:0/3103647658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 msgr2=0x55e428e53430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f5522d9d640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 0x7f550c0a6410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f551c005fd0 tx=0x7f551c009590 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5514009900 tx=0x7f5514005dc0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 -- 172.21.15.105:0/3103647658 shutdown_connections 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4ae80 0x55e428e53430 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.993+0000 7f552b7dc740 1 --2- 172.21.15.105:0/3103647658 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e4a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- 172.21.15.105:0/3103647658 >> 172.21.15.105:0/3103647658 conn(0x55e428e44ba0 msgr2=0x55e428e44f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- 172.21.15.105:0/3103647658 shutdown_connections 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- 172.21.15.105:0/3103647658 wait complete. 2024-04-22T06:38:47.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- start start 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 Processor -- start 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 0x55e428e531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 0x55e428e50550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e428e569b0 con 0x55e428e53a00 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e428d68500 con 0x55e428e4a4d0 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e428e56b30 con 0x55e428e4ae80 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f552b7dc740 10 client.0 _ll_register_callbacks cb 0x55e428d36b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.994+0000 7f5522d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5522d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57520/0 (socket says 172.21.15.105:57520) 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5522d9d640 1 -- 172.21.15.105:0/1246676560 learned_addr learned my addr 172.21.15.105:0/1246676560 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 0x55e428e50550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f552259c640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 0x55e428e531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 -- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 msgr2=0x55e428e531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 0x55e428e531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 -- 172.21.15.105:0/1246676560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 msgr2=0x55e428e50a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 -- 172.21.15.105:0/1246676560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e428ac13f0 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f552259c640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 0x55e428e531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5522d9d640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5521d9b640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 0x55e428e50550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55180ce4d0 tx=0x7f551809c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57db09b37b49091a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55180cee60 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551402cc90 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.995+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f55180d30d0 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f551809d420 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 10 client.25525 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e428b8b7d0 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551402cc90 con 0x55e428e4ae80 2024-04-22T06:38:47.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f55180a5500 con 0x55e428e4ae80 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 10 client.25525 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 10 client.25525 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 20 client.25525 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428e63b10 0x55e428e65f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 4 client.25525 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 --> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e428d4e930 con 0x55e428e63b10 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.996+0000 7f552259c640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428e63b10 0x55e428e65f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.997+0000 7f5509ffb640 10 client.25525 ms_handle_connect on v2:172.21.15.112:6834/3883368202 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.997+0000 7f552259c640 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428e63b10 0x55e428e65f00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=843317905bd5d2c5 server_cookie=2767338cb33677d4 in_seq=0 out_seq=0 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f5509ffb640 1 -- 172.21.15.105:0/1246676560 <== mds.1 v2:172.21.15.112:6834/3883368202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e428d4e930 con 0x55e428e63b10 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f5509ffb640 10 client.25525 handle_command_reply: tid=0 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 client.25525 shutdown 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 msgr2=0x55e428e50550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 0x55e428e50550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f55180ce4d0 tx=0x7f551809c920 comp rx=0 tx=0).stop 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 shutdown_connections 2024-04-22T06:38:48.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428e63b10 0x55e428e65f00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e53a00 0x55e428e50a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f5509ffb640 0 client.25525 ms_handle_reset on v2:172.21.15.112:6834/3883368202 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4ae80 0x55e428e50550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 --2- 172.21.15.105:0/1246676560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e4a4d0 0x55e428e531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 >> 172.21.15.105:0/1246676560 conn(0x55e428e44ba0 msgr2=0x55e428e468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 shutdown_connections 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 1 -- 172.21.15.105:0/1246676560 wait complete. 2024-04-22T06:38:48.001 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:47.998+0000 7f552b7dc740 20 client.25525 trim_cache size 0 max 16384 2024-04-22T06:38:48.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 Processor -- start 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 -- start start 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e5f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e52c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55241abb30 con 0x55e428e4ae80 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ad670 con 0x55e428e53a00 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552b7dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ac200 con 0x55e428e4a4d0 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57522/0 (socket says 172.21.15.105:57522) 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f5521d9b640 1 -- 172.21.15.105:0/4228237435 learned_addr learned my addr 172.21.15.105:0/4228237435 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f552259c640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e5f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.023+0000 7f5522d9d640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e52c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 -- 172.21.15.105:0/4228237435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 msgr2=0x55e428e52c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e52c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.023 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 -- 172.21.15.105:0/4228237435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 msgr2=0x55e428e43a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 -- 172.21.15.105:0/4228237435 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e428e449d0 con 0x55e428e4a4d0 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f5522d9d640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e52c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f5521d9b640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f552259c640 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e5f920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55e428e45c10 tx=0x7f5514005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966547ba4cfd9139 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f550bfff640 1 -- 172.21.15.105:0/4228237435 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514044040 con 0x55e428e4a4d0 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f550bfff640 1 -- 172.21.15.105:0/4228237435 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551402ca00 con 0x55e428e4a4d0 2024-04-22T06:38:48.024 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.024+0000 7f550bfff640 1 -- 172.21.15.105:0/4228237435 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514053820 con 0x55e428e4a4d0 2024-04-22T06:38:48.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- 172.21.15.105:0/4228237435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 msgr2=0x55e428e5f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e5f920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x55e428e45c10 tx=0x7f5514005280 comp rx=0 tx=0).stop 2024-04-22T06:38:48.027 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- 172.21.15.105:0/4228237435 shutdown_connections 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e52c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428e43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 --2- 172.21.15.105:0/4228237435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428e5f920 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- 172.21.15.105:0/4228237435 >> 172.21.15.105:0/4228237435 conn(0x55e428e44ba0 msgr2=0x55e428e43370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- 172.21.15.105:0/4228237435 shutdown_connections 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- 172.21.15.105:0/4228237435 wait complete. 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 -- start start 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.025+0000 7f552b7dc740 1 Processor -- start 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428ec21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e5b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f55241acfa0 con 0x55e428e4ae80 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ac8d0 con 0x55e428e53a00 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f55241a9b50 con 0x55e428e4a4d0 2024-04-22T06:38:48.028 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552b7dc740 10 client.0 _ll_register_callbacks cb 0x7f551809caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57528/0 (socket says 172.21.15.105:57528) 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 -- 172.21.15.105:0/741812339 learned_addr learned my addr 172.21.15.105:0/741812339 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f552259c640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428ec21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 -- 172.21.15.105:0/741812339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 msgr2=0x55e428ec21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428ec21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 -- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 msgr2=0x55e428e5b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5522d9d640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e5b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e5b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 -- 172.21.15.105:0/741812339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e428e5be20 con 0x55e428e4ae80 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5521d9b640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f551800cfd0 tx=0x7f55180bd1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211271ca7f902942 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.026+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55180cee60 con 0x55e428e4ae80 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f551809c380 con 0x55e428e4ae80 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551402ca00 con 0x55e428e4ae80 2024-04-22T06:38:48.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f55180b62a0 con 0x55e428e4ae80 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 10 client.16203 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e428ec9aa0 con 0x55e428e4ae80 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551402ca00 con 0x55e428e4ae80 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f55180a5be0 con 0x55e428e4ae80 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 10 client.16203 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 10 client.16203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.e 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 20 client.16203 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428eca0c0 0x55e428ecc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 4 client.16203 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 --> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e428dbcc10 con 0x55e428eca0c0 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.027+0000 7f552259c640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428eca0c0 0x55e428ecc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.030 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f5509ffb640 10 client.16203 ms_handle_connect on v2:172.21.15.112:6834/3883368202 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552259c640 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428eca0c0 0x55e428ecc4e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25e06baa49f7c74a server_cookie=c50a3656aec5531b in_seq=0 out_seq=0 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f5509ffb640 1 -- 172.21.15.105:0/741812339 <== mds.1 v2:172.21.15.112:6834/3883368202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e428dbcc10 con 0x55e428eca0c0 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f5509ffb640 10 client.16203 handle_command_reply: tid=0 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 client.16203 shutdown 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 msgr2=0x55e428ec2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f551800cfd0 tx=0x7f55180bd1d0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 shutdown_connections 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:6834/3883368202,v1:172.21.15.112:6835/3883368202] conn(0x55e428eca0c0 0x55e428ecc4e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e428e53a00 0x55e428e5b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e428e4ae80 0x55e428ec2730 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f5509ffb640 0 client.16203 ms_handle_reset on v2:172.21.15.112:6834/3883368202 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 --2- 172.21.15.105:0/741812339 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e428e4a4d0 0x55e428ec21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 >> 172.21.15.105:0/741812339 conn(0x55e428e44ba0 msgr2=0x55e428e45c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 shutdown_connections 2024-04-22T06:38:48.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 1 -- 172.21.15.105:0/741812339 wait complete. 2024-04-22T06:38:48.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.028+0000 7f552b7dc740 20 client.16203 trim_cache size 0 max 16384 2024-04-22T06:38:48.032 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:48.032 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 msgr2=0x7f550c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 0x7f550c0a6410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f551c005fd0 tx=0x7f551c009590 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 msgr2=0x7f552419e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f551800e9c0 tx=0x7f551800ee80 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 shutdown_connections 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f550c0a3f50 0x7f550c0a6410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f55241985c0 0x7f5524198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5524107260 0x7f552419e950 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 --2- 172.21.15.105:0/238340047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5524102190 0x7f552419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 >> 172.21.15.105:0/238340047 conn(0x7f55240793a0 msgr2=0x7f5524100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 shutdown_connections 2024-04-22T06:38:48.033 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.029+0000 7f5528bef640 1 -- 172.21.15.105:0/238340047 wait complete. 2024-04-22T06:38:48.039 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 -- 172.21.15.105:0/2571405215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1808101a40 msgr2=0x7f1808105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 --2- 172.21.15.105:0/2571405215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1808101a40 0x7f1808105a80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009900 tx=0x7f17f002bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 -- 172.21.15.105:0/2571405215 shutdown_connections 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 --2- 172.21.15.105:0/2571405215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 0x7f180810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 --2- 172.21.15.105:0/2571405215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1808101a40 0x7f1808105a80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 --2- 172.21.15.105:0/2571405215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f1808101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 -- 172.21.15.105:0/2571405215 >> 172.21.15.105:0/2571405215 conn(0x7f1808077f50 msgr2=0x7f1808078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 -- 172.21.15.105:0/2571405215 shutdown_connections 2024-04-22T06:38:48.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.149+0000 7f180cdb2640 1 -- 172.21.15.105:0/2571405215 wait complete. 2024-04-22T06:38:48.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 Processor -- start 2024-04-22T06:38:48.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 -- start start 2024-04-22T06:38:48.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f18081987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 0x7f1808198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18081110a0 con 0x7f180819d010 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1808110f20 con 0x7f1808106160 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f180cdb2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f1808111220 con 0x7f1808101090 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57538/0 (socket says 172.21.15.105:57538) 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.150+0000 7f1806d9d640 1 -- 172.21.15.105:0/897670648 learned_addr learned my addr 172.21.15.105:0/897670648 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f180659c640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f18081987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f17fffff640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 0x7f1808198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 -- 172.21.15.105:0/897670648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 msgr2=0x7f18081987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f18081987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 -- 172.21.15.105:0/897670648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 msgr2=0x7f1808198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 0x7f1808198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 -- 172.21.15.105:0/897670648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180819db00 con 0x7f180819d010 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f180659c640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f18081987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:48.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f1806d9d640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f180000eaa0 tx=0x7f180000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f17ff7fe640 1 -- 172.21.15.105:0/897670648 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180000cdb0 con 0x7f180819d010 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180819ddf0 con 0x7f180819d010 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f17ff7fe640 1 -- 172.21.15.105:0/897670648 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1800005020 con 0x7f180819d010 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f17ff7fe640 1 -- 172.21.15.105:0/897670648 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18000116d0 con 0x7f180819d010 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.151+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081a55b0 con 0x7f180819d010 2024-04-22T06:38:48.152 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.154+0000 7f17ff7fe640 1 -- 172.21.15.105:0/897670648 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f18000051d0 con 0x7f180819d010 2024-04-22T06:38:48.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.154+0000 7f17ff7fe640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 0x7f17f40a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.154+0000 7f180f9a7740 1 Processor -- start 2024-04-22T06:38:48.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180659c640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 0x7f17f40a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f17ff7fe640 1 -- 172.21.15.105:0/897670648 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f18000d0c40 con 0x7f180819d010 2024-04-22T06:38:48.153 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 -- start start 2024-04-22T06:38:48.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c13932cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.161 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c1393b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557c138931f0 con 0x557c1393be10 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557c138304b0 con 0x557c139328e0 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180f9a7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557c1388f1b0 con 0x557c13933290 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180659c640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 0x7f17f40a6410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009fc0 tx=0x7f17f8005f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57552/0 (socket says 172.21.15.105:57552) 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 -- 172.21.15.105:0/834252835 learned_addr learned my addr 172.21.15.105:0/834252835 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f17fffff640 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c1393b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f180659c640 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c13932cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 -- 172.21.15.105:0/834252835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 msgr2=0x557c1393b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.162 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c1393b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 -- 172.21.15.105:0/834252835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 msgr2=0x557c13932cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.155+0000 7f1806d9d640 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c13932cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.156+0000 7f1806d9d640 1 -- 172.21.15.105:0/834252835 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c138221b0 con 0x557c1393be10 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.156+0000 7f1806d9d640 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f180000efd0 tx=0x7f180009c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf92f22192643587 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.156+0000 7f17fcff9640 1 -- 172.21.15.105:0/834252835 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180001f040 con 0x557c1393be10 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.156+0000 7f17fcff9640 1 -- 172.21.15.105:0/834252835 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f180009ac60 con 0x557c1393be10 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.156+0000 7f17fcff9640 1 -- 172.21.15.105:0/834252835 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18000872e0 con 0x557c1393be10 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- 172.21.15.105:0/834252835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 msgr2=0x557c1393e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f180000efd0 tx=0x7f180009c5d0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- 172.21.15.105:0/834252835 shutdown_connections 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c1393e260 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c1393b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 --2- 172.21.15.105:0/834252835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c13932cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- 172.21.15.105:0/834252835 >> 172.21.15.105:0/834252835 conn(0x557c1392cfb0 msgr2=0x557c1392d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.163 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- 172.21.15.105:0/834252835 shutdown_connections 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- 172.21.15.105:0/834252835 wait complete. 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.157+0000 7f180f9a7740 1 -- start start 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 Processor -- start 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 0x557c139352a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c139357e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557c139286d0 con 0x557c139328e0 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557c1384c500 con 0x557c13933290 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557c13928850 con 0x557c1393be10 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180f9a7740 10 client.0 _ll_register_callbacks cb 0x557c1381ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57568/0 (socket says 172.21.15.105:57568) 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 -- 172.21.15.105:0/1821527985 learned_addr learned my addr 172.21.15.105:0/1821527985 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f1806d9d640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c139357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.164 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f17fffff640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 0x557c139352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 -- 172.21.15.105:0/1821527985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 msgr2=0x557c139357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c139357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 -- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 msgr2=0x557c139352a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 0x557c139352a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f180659c640 1 -- 172.21.15.105:0/1821527985 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c135a53f0 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f1806d9d640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c139357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.158+0000 7f17fffff640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 0x557c139352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f180659c640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f17f8003680 tx=0x7f17f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23752aab34dd85e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f17edffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f800fa50 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f17f800fbf0 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f180009ac60 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17f800fdf0 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f180f9a7740 10 client.16215 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557c1366f7d0 con 0x557c139328e0 2024-04-22T06:38:48.165 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.159+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f180009ac60 con 0x557c139328e0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f17f8025070 con 0x557c139328e0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 10 client.16215 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 20 client.16215 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139503e0 0x557c13952830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 4 client.16215 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c13832930 con 0x557c139503e0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f17fffff640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139503e0 0x557c13952830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f17edffb640 10 client.16215 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.160+0000 7f17fffff640 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139503e0 0x557c13952830 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fca3f564ee14d6b server_cookie=330c663f19e882df in_seq=0 out_seq=0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f17edffb640 1 -- 172.21.15.105:0/1821527985 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c13832930 con 0x557c139503e0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f17edffb640 10 client.16215 handle_command_reply: tid=0 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 client.16215 shutdown 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 msgr2=0x557c13934d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f17f8003680 tx=0x7f17f8004730 comp rx=0 tx=0).stop 2024-04-22T06:38:48.166 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 shutdown_connections 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139503e0 0x557c13952830 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f17edffb640 0 client.16215 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c139357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c13933290 0x557c139352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 --2- 172.21.15.105:0/1821527985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c139328e0 0x557c13934d60 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 >> 172.21.15.105:0/1821527985 conn(0x557c1392cfb0 msgr2=0x557c1392ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.161+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 shutdown_connections 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.162+0000 7f180f9a7740 1 -- 172.21.15.105:0/1821527985 wait complete. 2024-04-22T06:38:48.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.162+0000 7f180f9a7740 20 client.16215 trim_cache size 0 max 16384 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 Processor -- start 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 -- start start 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1393b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c13935600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18081abb20 con 0x557c1393be10 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18081ad660 con 0x557c139328e0 2024-04-22T06:38:48.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180f9a7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18081ac1f0 con 0x557c13933290 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57578/0 (socket says 172.21.15.105:57578) 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 -- 172.21.15.105:0/3562921956 learned_addr learned my addr 172.21.15.105:0/3562921956 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180659c640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1393b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 -- 172.21.15.105:0/3562921956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 msgr2=0x557c13935600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f17fffff640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c13935600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c13935600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 -- 172.21.15.105:0/3562921956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 msgr2=0x557c1393b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1393b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f1806d9d640 1 -- 172.21.15.105:0/3562921956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c13924470 con 0x557c1393be10 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.179+0000 7f180659c640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1393b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.180+0000 7f1806d9d640 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f180009cb20 tx=0x7f1800005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd7f0f6ecbd6dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.180+0000 7f17effff640 1 -- 172.21.15.105:0/3562921956 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180001f040 con 0x557c1393be10 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.180+0000 7f17effff640 1 -- 172.21.15.105:0/3562921956 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1800098fd0 con 0x557c1393be10 2024-04-22T06:38:48.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.180+0000 7f17effff640 1 -- 172.21.15.105:0/3562921956 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800087320 con 0x557c1393be10 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- 172.21.15.105:0/3562921956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 msgr2=0x557c13958100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f180009cb20 tx=0x7f1800005890 comp rx=0 tx=0).stop 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- 172.21.15.105:0/3562921956 shutdown_connections 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c1393be10 0x557c13958100 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c13933290 0x557c13935600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- 172.21.15.105:0/3562921956 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1393b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- 172.21.15.105:0/3562921956 >> 172.21.15.105:0/3562921956 conn(0x557c1392cfb0 msgr2=0x557c13938c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- 172.21.15.105:0/3562921956 shutdown_connections 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- 172.21.15.105:0/3562921956 wait complete. 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- start start 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 Processor -- start 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1394ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c1392b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.181+0000 7f180f9a7740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18081acf90 con 0x557c13933290 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180f9a7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f18081ac8c0 con 0x557c139328e0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180f9a7740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f18081a9b20 con 0x557c1393be10 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180f9a7740 10 client.0 _ll_register_callbacks cb 0x7f17f80044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17fffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17fffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57594/0 (socket says 172.21.15.105:57594) 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17fffff640 1 -- 172.21.15.105:0/2827198696 learned_addr learned my addr 172.21.15.105:0/2827198696 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1394ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f1806d9d640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c1392b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 -- 172.21.15.105:0/2827198696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 msgr2=0x557c1392b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c1392b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 -- 172.21.15.105:0/2827198696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 msgr2=0x557c1392b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 -- 172.21.15.105:0/2827198696 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c139b5970 con 0x557c139328e0 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17fffff640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:48.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180659c640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1394ce00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17f8026760 tx=0x7f17f8027dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13cfb3f9035758f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17edffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f800fa60 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1800098fd0 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.182+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f17f801d3f0 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17f800fc00 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 10 client.25532 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557c139b6ac0 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1800098fd0 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f17f8039030 con 0x557c139328e0 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 10 client.25532 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 10 client.25532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 20 client.25532 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139b7230 0x557c139b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 4 client.25532 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-22T06:38:48.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c138a0c10 con 0x557c139b7230 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.183+0000 7f17fffff640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139b7230 0x557c139b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f17edffb640 10 client.25532 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f17fffff640 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139b7230 0x557c139b9650 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=608381eeb18245af server_cookie=98883994ccda4c13 in_seq=0 out_seq=0 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f17edffb640 1 -- 172.21.15.105:0/2827198696 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c138a0c10 con 0x557c139b7230 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f17edffb640 10 client.25532 handle_command_reply: tid=0 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 client.25532 shutdown 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 msgr2=0x557c1394ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1394ce00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17f8026760 tx=0x7f17f8027dd0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 shutdown_connections 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557c139b7230 0x557c139b9650 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557c1393be10 0x557c1392b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f17edffb640 0 client.25532 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557c13933290 0x557c1392b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 --2- 172.21.15.105:0/2827198696 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557c139328e0 0x557c1394ce00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 >> 172.21.15.105:0/2827198696 conn(0x557c1392cfb0 msgr2=0x557c13938370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 shutdown_connections 2024-04-22T06:38:48.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 1 -- 172.21.15.105:0/2827198696 wait complete. 2024-04-22T06:38:48.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.184+0000 7f180f9a7740 20 client.25532 trim_cache size 0 max 16384 2024-04-22T06:38:48.188 INFO:teuthology.orchestra.run.smithi105.stdout:[] 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 msgr2=0x7f17f40a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 0x7f17f40a6410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009fc0 tx=0x7f17f8005f90 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 msgr2=0x7f180819d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f180000eaa0 tx=0x7f180000ef60 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 shutdown_connections 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f17f40a3f50 0x7f17f40a6410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f180819d010 0x7f180819d4c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1808106160 0x7f1808198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 --2- 172.21.15.105:0/897670648 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1808101090 0x7f18081987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 >> 172.21.15.105:0/897670648 conn(0x7f1808077f50 msgr2=0x7f18080ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 shutdown_connections 2024-04-22T06:38:48.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:48.185+0000 7f180cdb2640 1 -- 172.21.15.105:0/897670648 wait complete. 2024-04-22T06:38:48.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:48 smithi177 ceph-mon[31346]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-22T06:38:48.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:48 smithi105 ceph-mon[27916]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-22T06:38:48.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:48 smithi112 ceph-mon[31236]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-22T06:38:49.198 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-22T06:38:49.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:38:49.198 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.321+0000 7f54dcec8640 1 -- 172.21.15.105:0/2810057564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d8101a40 msgr2=0x7f54d8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.321+0000 7f54dcec8640 1 --2- 172.21.15.105:0/2810057564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d8101a40 0x7f54d8105a80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009900 tx=0x7f54cc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 -- 172.21.15.105:0/2810057564 shutdown_connections 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 --2- 172.21.15.105:0/2810057564 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8106160 0x7f54d810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 --2- 172.21.15.105:0/2810057564 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d8101a40 0x7f54d8105a80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 --2- 172.21.15.105:0/2810057564 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8101090 0x7f54d8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 -- 172.21.15.105:0/2810057564 >> 172.21.15.105:0/2810057564 conn(0x7f54d8077f50 msgr2=0x7f54d8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 -- 172.21.15.105:0/2810057564 shutdown_connections 2024-04-22T06:38:49.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 -- 172.21.15.105:0/2810057564 wait complete. 2024-04-22T06:38:49.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.322+0000 7f54dcec8640 1 Processor -- start 2024-04-22T06:38:49.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 -- start start 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8101090 0x7f54d810e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8106160 0x7f54d810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54d8112200 con 0x7f54d810f140 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f54d8112080 con 0x7f54d8101090 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54dcec8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54d8112380 con 0x7f54d8106160 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57608/0 (socket says 172.21.15.105:57608) 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 -- 172.21.15.105:0/3123162131 learned_addr learned my addr 172.21.15.105:0/3123162131 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d659c640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8101090 0x7f54d810e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8106160 0x7f54d810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 -- 172.21.15.105:0/3123162131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8106160 msgr2=0x7f54d810ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8106160 0x7f54d810ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 -- 172.21.15.105:0/3123162131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8101090 msgr2=0x7f54d810e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8101090 0x7f54d810e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.323+0000 7f54d6d9d640 1 -- 172.21.15.105:0/3123162131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d81a7090 con 0x7f54d810f140 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.324+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f54d000eaa0 tx=0x7f54d000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.324+0000 7f54c77fe640 1 -- 172.21.15.105:0/3123162131 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0004d20 con 0x7f54d810f140 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.324+0000 7f54c77fe640 1 -- 172.21.15.105:0/3123162131 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d0005020 con 0x7f54d810f140 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.324+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d81a7380 con 0x7f54d810f140 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.325+0000 7f54c77fe640 1 -- 172.21.15.105:0/3123162131 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d00116b0 con 0x7f54d810f140 2024-04-22T06:38:49.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.325+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d81a7850 con 0x7f54d810f140 2024-04-22T06:38:49.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.327+0000 7f54c77fe640 1 -- 172.21.15.105:0/3123162131 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f54d00051d0 con 0x7f54d810f140 2024-04-22T06:38:49.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54c77fe640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 0x7f54c00a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54d659c640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 0x7f54c00a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54dfb97740 1 Processor -- start 2024-04-22T06:38:49.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54c77fe640 1 -- 172.21.15.105:0/3123162131 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f54d00d0d80 con 0x7f54d810f140 2024-04-22T06:38:49.328 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54dfb97740 1 -- start start 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.328+0000 7f54d659c640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 0x7f54c00a63c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009990 tx=0x7f54c8009110 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69f290 0x55769f6a7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f6a7e10 0x55769f6aa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55769f602f00 con 0x55769f69e8e0 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55769f5a0590 con 0x55769f69f290 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54dfb97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55769f5feec0 con 0x55769f6a7e10 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57622/0 (socket says 172.21.15.105:57622) 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 -- 172.21.15.105:0/2201885448 learned_addr learned my addr 172.21.15.105:0/2201885448 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69f290 0x55769f6a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f6a7e10 0x55769f6aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 -- 172.21.15.105:0/2201885448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f6a7e10 msgr2=0x55769f6aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f6a7e10 0x55769f6aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 -- 172.21.15.105:0/2201885448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69f290 msgr2=0x55769f6a7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69f290 0x55769f6a7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 -- 172.21.15.105:0/2201885448 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55769f3dd7e0 con 0x55769f69e8e0 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54d659c640 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009fd0 tx=0x7f54c8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2f50934feba5e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54c4ff9640 1 -- 172.21.15.105:0/2201885448 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c801b040 con 0x55769f69e8e0 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.329+0000 7f54c4ff9640 1 -- 172.21.15.105:0/2201885448 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c8024070 con 0x55769f69e8e0 2024-04-22T06:38:49.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54c4ff9640 1 -- 172.21.15.105:0/2201885448 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c801f3f0 con 0x55769f69e8e0 2024-04-22T06:38:49.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 -- 172.21.15.105:0/2201885448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 msgr2=0x55769f69ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009fd0 tx=0x7f54c8011000 comp rx=0 tx=0).stop 2024-04-22T06:38:49.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 -- 172.21.15.105:0/2201885448 shutdown_connections 2024-04-22T06:38:49.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f6a7e10 0x55769f6aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69f290 0x55769f6a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 --2- 172.21.15.105:0/2201885448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f69ecc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.330+0000 7f54dfb97740 1 -- 172.21.15.105:0/2201885448 >> 172.21.15.105:0/2201885448 conn(0x55769f698fb0 msgr2=0x55769f699390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- 172.21.15.105:0/2201885448 shutdown_connections 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- 172.21.15.105:0/2201885448 wait complete. 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- start start 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 Processor -- start 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 0x55769f6a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6a5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55769f6aad80 con 0x55769f69f290 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55769f5bc580 con 0x55769f6a7e10 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55769f6aaf00 con 0x55769f69e8e0 2024-04-22T06:38:49.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54dfb97740 10 client.0 _ll_register_callbacks cb 0x55769f5c2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.331+0000 7f54d5d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57638/0 (socket says 172.21.15.105:57638) 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 -- 172.21.15.105:0/179009577 learned_addr learned my addr 172.21.15.105:0/179009577 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d659c640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 0x55769f6a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 -- 172.21.15.105:0/179009577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 msgr2=0x55769f6a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 0x55769f6a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 -- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 msgr2=0x55769f6a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6a5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 -- 172.21.15.105:0/179009577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55769f43df70 con 0x55769f69f290 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6a5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d659c640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 0x55769f6a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:49.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f54d806b6e0 tx=0x7f54cc02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3029447b09f97141 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc02c610 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f54cc005030 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.332+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c8024070 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54cc005230 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 10 client.16233 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55769f2d5170 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54cc0053d0 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f54cc0038a0 con 0x55769f69f290 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 10 client.16233 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 10 client.16233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:49.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 20 client.16233 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f6b8370 0x55769f6ba7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 4 client.16233 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.333+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f54c8024070 con 0x55769f6b8370 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.334+0000 7f54d659c640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f6b8370 0x55769f6ba7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.334+0000 7f54d659c640 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f6b8370 0x55769f6ba7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4545845c9901c66 server_cookie=7d0ab1e61e2228a3 in_seq=0 out_seq=0 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.334+0000 7f54bdffb640 10 client.16233 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54bdffb640 1 -- 172.21.15.105:0/179009577 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f54c8024070 con 0x55769f6b8370 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54bdffb640 10 client.16233 handle_command_reply: tid=0 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 client.16233 shutdown 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 msgr2=0x55769f6a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f54d806b6e0 tx=0x7f54cc02c9f0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 shutdown_connections 2024-04-22T06:38:49.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f6b8370 0x55769f6ba7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6a5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69f290 0x55769f6a5350 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 --2- 172.21.15.105:0/179009577 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69e8e0 0x55769f6a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 >> 172.21.15.105:0/179009577 conn(0x55769f698fb0 msgr2=0x55769f69ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54bdffb640 0 client.16233 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 shutdown_connections 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 1 -- 172.21.15.105:0/179009577 wait complete. 2024-04-22T06:38:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.335+0000 7f54dfb97740 20 client.16233 trim_cache size 0 max 16384 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 Processor -- start 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 -- start start 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81aff10 con 0x55769f69e8e0 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81b1a50 con 0x55769f6a7e10 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.368+0000 7f54dfb97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81b05e0 con 0x55769f69f290 2024-04-22T06:38:49.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57650/0 (socket says 172.21.15.105:57650) 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 -- 172.21.15.105:0/3326513137 learned_addr learned my addr 172.21.15.105:0/3326513137 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 -- 172.21.15.105:0/3326513137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 msgr2=0x55769f6b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 -- 172.21.15.105:0/3326513137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 msgr2=0x55769f6c00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6c00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 -- 172.21.15.105:0/3326513137 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55769f6a3360 con 0x55769f69e8e0 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:49.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54d659c640 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009fd0 tx=0x7f54c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c194dc2f24103 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54bffff640 1 -- 172.21.15.105:0/3326513137 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8033bd0 con 0x55769f69e8e0 2024-04-22T06:38:49.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54bffff640 1 -- 172.21.15.105:0/3326513137 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c802e070 con 0x55769f69e8e0 2024-04-22T06:38:49.370 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.369+0000 7f54bffff640 1 -- 172.21.15.105:0/3326513137 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8033850 con 0x55769f69e8e0 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 -- 172.21.15.105:0/3326513137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 msgr2=0x55769f6a7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009fd0 tx=0x7f54c80046c0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 -- 172.21.15.105:0/3326513137 shutdown_connections 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f6a7e10 0x55769f6c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 --2- 172.21.15.105:0/3326513137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f69e8e0 0x55769f6a7540 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 -- 172.21.15.105:0/3326513137 >> 172.21.15.105:0/3326513137 conn(0x55769f698fb0 msgr2=0x55769f6a36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 -- 172.21.15.105:0/3326513137 shutdown_connections 2024-04-22T06:38:49.374 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.370+0000 7f54dfb97740 1 -- 172.21.15.105:0/3326513137 wait complete. 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 -- start start 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 Processor -- start 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 0x55769f6b0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81b1380 con 0x55769f6a7e10 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81b0cb0 con 0x55769f69e8e0 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81adf30 con 0x55769f69f290 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54dfb97740 10 client.0 _ll_register_callbacks cb 0x7f54cc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57654/0 (socket says 172.21.15.105:57654) 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 -- 172.21.15.105:0/39031322 learned_addr learned my addr 172.21.15.105:0/39031322 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d5d9b640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.375 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d659c640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 0x55769f6b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 -- 172.21.15.105:0/39031322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 msgr2=0x55769f6b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 -- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 msgr2=0x55769f6b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 0x55769f6b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d6d9d640 1 -- 172.21.15.105:0/39031322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55769f6a1e20 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.371+0000 7f54d659c640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 0x55769f6b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54d6d9d640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f54d009c6f0 tx=0x7f54d009c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a810ac7a1d1702d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d001f040 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f54d00bbdb0 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c802e070 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54d00a5660 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 10 client.16239 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55769f71e540 con 0x55769f6a7e10 2024-04-22T06:38:49.376 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54c802e070 con 0x55769f6a7e10 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f54d000f040 con 0x55769f6a7e10 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 10 client.16239 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 20 client.16239 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f71ecb0 0x55769f7210d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 4 client.16239 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.372+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55769f610920 con 0x55769f71ecb0 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.373+0000 7f54d659c640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f71ecb0 0x55769f7210d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.373+0000 7f54d659c640 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f71ecb0 0x55769f7210d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec047b2f04586cd1 server_cookie=58d3272fba184596 in_seq=0 out_seq=0 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.373+0000 7f54bdffb640 10 client.16239 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54bdffb640 1 -- 172.21.15.105:0/39031322 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55769f610920 con 0x55769f71ecb0 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54bdffb640 10 client.16239 handle_command_reply: tid=0 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 client.16239 shutdown 2024-04-22T06:38:49.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 msgr2=0x55769f6a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f54d009c6f0 tx=0x7f54d009c450 comp rx=0 tx=0).stop 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 shutdown_connections 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55769f71ecb0 0x55769f7210d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54bdffb640 0 client.16239 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55769f6a7e10 0x55769f6a1620 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55769f69f290 0x55769f6b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 --2- 172.21.15.105:0/39031322 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55769f69e8e0 0x55769f6b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.374+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 >> 172.21.15.105:0/39031322 conn(0x55769f698fb0 msgr2=0x55769f69a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 shutdown_connections 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dfb97740 1 -- 172.21.15.105:0/39031322 wait complete. 2024-04-22T06:38:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dfb97740 20 client.16239 trim_cache size 0 max 16384 2024-04-22T06:38:49.379 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "return_code": 0, 2024-04-22T06:38:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_tag": "1dc9ba3f-9cb9-4d1d-a95d-73a068cfc70f", 2024-04-22T06:38:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": "asynchronous" 2024-04-22T06:38:49.379 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 msgr2=0x7f54c00a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 0x7f54c00a63c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009990 tx=0x7f54c8009110 comp rx=0 tx=0).stop 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 msgr2=0x7f54d81a6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f54d000eaa0 tx=0x7f54d000eea0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 shutdown_connections 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f54c00a3f00 0x7f54c00a63c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f54d810f140 0x7f54d81a6990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54d8106160 0x7f54d810ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 --2- 172.21.15.105:0/3123162131 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54d8101090 0x7f54d810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 >> 172.21.15.105:0/3123162131 conn(0x7f54d8077f50 msgr2=0x7f54d80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 shutdown_connections 2024-04-22T06:38:49.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.375+0000 7f54dcec8640 1 -- 172.21.15.105:0/3123162131 wait complete. 2024-04-22T06:38:49.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:38:49.388 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 -- 172.21.15.105:0/3633913741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c106150 msgr2=0x7fb69c10dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/3633913741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c106150 0x7fb69c10dd20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb694009ca0 tx=0x7fb69402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 -- 172.21.15.105:0/3633913741 shutdown_connections 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/3633913741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c106150 0x7fb69c10dd20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/3633913741 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c101a30 0x7fb69c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/3633913741 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c101080 0x7fb69c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 -- 172.21.15.105:0/3633913741 >> 172.21.15.105:0/3633913741 conn(0x7fb69c078000 msgr2=0x7fb69c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.495 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.496+0000 7fb6a0be2640 1 -- 172.21.15.105:0/3633913741 shutdown_connections 2024-04-22T06:38:49.496 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.497+0000 7fb6a0be2640 1 -- 172.21.15.105:0/3633913741 wait complete. 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.497+0000 7fb6a0be2640 1 Processor -- start 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.497+0000 7fb6a0be2640 1 -- start start 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 0x7fb69c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 0x7fb69c19c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c1110a0 con 0x7fb69c101080 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c110f20 con 0x7fb69c19bd60 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb6a0be2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c111220 con 0x7fb69c106150 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57682/0 (socket says 172.21.15.105:57682) 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 -- 172.21.15.105:0/112573143 learned_addr learned my addr 172.21.15.105:0/112573143 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.498 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb699d9b640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 0x7fb69c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 0x7fb69c19c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 -- 172.21.15.105:0/112573143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 msgr2=0x7fb69c1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 0x7fb69c1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 -- 172.21.15.105:0/112573143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 msgr2=0x7fb69c19c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 0x7fb69c19c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 -- 172.21.15.105:0/112573143 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c19c850 con 0x7fb69c101080 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb699d9b640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 0x7fb69c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 0x7fb69c19c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.498+0000 7fb69a59c640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c920 tx=0x7fb68c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.499+0000 7fb68b7fe640 1 -- 172.21.15.105:0/112573143 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c007c70 con 0x7fb69c101080 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.499+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c19cb40 con 0x7fb69c101080 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.499+0000 7fb68b7fe640 1 -- 172.21.15.105:0/112573143 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb68c007e10 con 0x7fb69c101080 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.499+0000 7fb68b7fe640 1 -- 172.21.15.105:0/112573143 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c018820 con 0x7fb69c101080 2024-04-22T06:38:49.499 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.499+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c1a4330 con 0x7fb69c101080 2024-04-22T06:38:49.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.501+0000 7fb68b7fe640 1 -- 172.21.15.105:0/112573143 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fb68c00fa90 con 0x7fb69c101080 2024-04-22T06:38:49.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.502+0000 7fb68b7fe640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 0x7fb6840a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.502+0000 7fb699d9b640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 0x7fb6840a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.502+0000 7fb68b7fe640 1 -- 172.21.15.105:0/112573143 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb68c0d0d80 con 0x7fb69c101080 2024-04-22T06:38:49.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.502+0000 7fb699d9b640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 0x7fb6840a6390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb690005e10 tx=0x7fb690005da0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.502+0000 7fb6a37cb740 1 Processor -- start 2024-04-22T06:38:49.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 -- start start 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b995750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b998080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57690/0 (socket says 172.21.15.105:57690) 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb69a59c640 1 -- 172.21.15.105:0/1658616795 learned_addr learned my addr 172.21.15.105:0/1658616795 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55811b8ef1e0 con 0x55811b989f60 2024-04-22T06:38:49.503 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55811b88c390 con 0x55811b995c90 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55811b8eb1a0 con 0x55811b98a910 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb699d9b640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b995750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.503+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b998080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 -- 172.21.15.105:0/1658616795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 msgr2=0x55811b995750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b995750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 -- 172.21.15.105:0/1658616795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 msgr2=0x55811b98a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 -- 172.21.15.105:0/1658616795 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55811b6cb7d0 con 0x55811b995c90 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69a59c640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b998080 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb69402c700 tx=0x7fb694009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec70c5528d960603 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb688ff9640 1 -- 172.21.15.105:0/1658616795 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694043020 con 0x55811b995c90 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb688ff9640 1 -- 172.21.15.105:0/1658616795 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb694032970 con 0x55811b995c90 2024-04-22T06:38:49.504 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.504+0000 7fb688ff9640 1 -- 172.21.15.105:0/1658616795 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb694039040 con 0x55811b995c90 2024-04-22T06:38:49.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 msgr2=0x55811b998080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.508 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b998080 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb69402c700 tx=0x7fb694009900 comp rx=0 tx=0).stop 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 shutdown_connections 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b998080 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b995750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/1658616795 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b989f60 0x55811b98a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 >> 172.21.15.105:0/1658616795 conn(0x55811b984cb0 msgr2=0x55811b985090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 shutdown_connections 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.505+0000 7fb6a37cb740 1 -- 172.21.15.105:0/1658616795 wait complete. 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 -- start start 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 Processor -- start 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b992c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b993690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55811b9989f0 con 0x55811b98a910 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55811b8a82f0 con 0x55811b995c90 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55811b998b70 con 0x55811b989f60 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57700/0 (socket says 172.21.15.105:57700) 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 -- 172.21.15.105:0/3161976039 learned_addr learned my addr 172.21.15.105:0/3161976039 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb6a37cb740 10 client.0 _ll_register_callbacks cb 0x55811b8b5580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b993690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b992c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 -- 172.21.15.105:0/3161976039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 msgr2=0x55811b992c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b992c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 -- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 msgr2=0x55811b993690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b993690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.506+0000 7fb699d9b640 1 -- 172.21.15.105:0/3161976039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55811b6013f0 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb690010a90 tx=0x7fb6900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b5dc2fe040fe7b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb681ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69001de10 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb69001e430 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb694032970 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb69001e630 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb6a37cb740 10 client.16251 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55811b5c10f0 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.507+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb694032970 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fb690026810 con 0x55811b98a910 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 10 client.16251 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:49.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 20 client.16251 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811b9a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 4 client.16251 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55811b8fcc00 con 0x55811b9a5b00 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811b9a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb681ffb640 10 client.16251 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.508+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811b9a7f30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8ede1de6f85e4d server_cookie=b12da497126d9b94 in_seq=0 out_seq=0 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb681ffb640 1 -- 172.21.15.105:0/3161976039 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55811b8fcc00 con 0x55811b9a5b00 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb681ffb640 10 client.16251 handle_command_reply: tid=0 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb6a37cb740 1 client.16251 shutdown 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 msgr2=0x55811b993150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb690010a90 tx=0x7fb6900046c0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 shutdown_connections 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.509+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811b9a7f30 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b993690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb681ffb640 0 client.16251 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.512 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b993150 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3161976039 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b992c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 >> 172.21.15.105:0/3161976039 conn(0x55811b984cb0 msgr2=0x55811b986a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 shutdown_connections 2024-04-22T06:38:49.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3161976039 wait complete. 2024-04-22T06:38:49.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.510+0000 7fb6a37cb740 20 client.16251 trim_cache size 0 max 16384 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.528+0000 7fb6a37cb740 1 Processor -- start 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 -- start start 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b995340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b9a1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c00e850 con 0x55811b98a910 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c010390 con 0x55811b995c90 2024-04-22T06:38:49.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c00ef20 con 0x55811b989f60 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57710/0 (socket says 172.21.15.105:57710) 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 -- 172.21.15.105:0/3142201790 learned_addr learned my addr 172.21.15.105:0/3142201790 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b995340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b9a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 -- 172.21.15.105:0/3142201790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 msgr2=0x55811b995340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b995340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 -- 172.21.15.105:0/3142201790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 msgr2=0x55811b9a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b9a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 -- 172.21.15.105:0/3142201790 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55811b989670 con 0x55811b98a910 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.529+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb690026e70 tx=0x7fb690000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd1e1983b1eabfc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb683fff640 1 -- 172.21.15.105:0/3142201790 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69003e020 con 0x55811b98a910 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb683fff640 1 -- 172.21.15.105:0/3142201790 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb69001d790 con 0x55811b98a910 2024-04-22T06:38:49.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb683fff640 1 -- 172.21.15.105:0/3142201790 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690026650 con 0x55811b98a910 2024-04-22T06:38:49.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3142201790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 msgr2=0x55811b991b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb690026e70 tx=0x7fb690000bc0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.533 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3142201790 shutdown_connections 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b995c90 0x55811b9a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b98a910 0x55811b991b50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3142201790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b989f60 0x55811b995340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3142201790 >> 172.21.15.105:0/3142201790 conn(0x55811b984cb0 msgr2=0x55811b986990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3142201790 shutdown_connections 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.530+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3142201790 wait complete. 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 -- start start 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 Processor -- start 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b99de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 0x55811b99e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c00fcc0 con 0x55811b995c90 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c00f5f0 con 0x55811b989f60 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c00c830 con 0x55811b98a910 2024-04-22T06:38:49.534 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb6a37cb740 10 client.0 _ll_register_callbacks cb 0x7fb690010710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb69ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 0x55811b99e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb69a59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:51514/0 (socket says 172.21.15.105:51514) 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb69a59c640 1 -- 172.21.15.105:0/3057049260 learned_addr learned my addr 172.21.15.105:0/3057049260 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.531+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b99de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 -- 172.21.15.105:0/3057049260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 msgr2=0x55811b99de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b99de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 -- 172.21.15.105:0/3057049260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 msgr2=0x55811b99e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 0x55811b99e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 -- 172.21.15.105:0/3057049260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55811b998080 con 0x55811b989f60 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69ad9d640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 0x55811b99e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb69a59c640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb68c09c6f0 tx=0x7fb68c087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baa8c55bd1c9ed40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb681ffb640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0b7070 con 0x55811b989f60 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb69001d790 con 0x55811b989f60 2024-04-22T06:38:49.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fb68c0a51d0 con 0x55811b989f60 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb68c0a53d0 con 0x55811b989f60 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb6a37cb740 10 client.25562 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55811b984ae0 con 0x55811b989f60 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.532+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb69001d790 con 0x55811b989f60 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fb68c09be40 con 0x55811b989f60 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 10 client.25562 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 10 client.25562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 20 client.25562 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811ba10da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 4 client.25562 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55811b8f6a70 con 0x55811b9a5b00 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811ba10da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb681ffb640 10 client.25562 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.536 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.533+0000 7fb699d9b640 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811ba10da0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba156915bef96e6d server_cookie=2af62559e0448962 in_seq=0 out_seq=0 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb681ffb640 1 -- 172.21.15.105:0/3057049260 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55811b8f6a70 con 0x55811b9a5b00 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb681ffb640 10 client.25562 handle_command_reply: tid=0 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 client.25562 shutdown 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 msgr2=0x55811ba07960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb68c09c6f0 tx=0x7fb68c087040 comp rx=0 tx=0).stop 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 shutdown_connections 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55811b9a5b00 0x55811ba10da0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55811b995c90 0x55811b99e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.537 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55811b98a910 0x55811b99de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 --2- 172.21.15.105:0/3057049260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55811b989f60 0x55811ba07960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb681ffb640 0 client.25562 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 >> 172.21.15.105:0/3057049260 conn(0x55811b984cb0 msgr2=0x55811b986100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 shutdown_connections 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 1 -- 172.21.15.105:0/3057049260 wait complete. 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a37cb740 20 client.25562 trim_cache size 0 max 16384 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": { 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "1dc9ba3f-9cb9-4d1d-a95d-73a068cfc70f": { 2024-04-22T06:38:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "path": "/", 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "tag": "1dc9ba3f-9cb9-4d1d-a95d-73a068cfc70f", 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "options": "recursive,force" 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:49.539 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 msgr2=0x7fb6840a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 0x7fb6840a6390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb690005e10 tx=0x7fb690005da0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 msgr2=0x7fb69c1974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.534+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c920 tx=0x7fb68c00cde0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 shutdown_connections 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6840a3ed0 0x7fb6840a6390 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb69c19bd60 0x7fb69c19c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb69c106150 0x7fb69c1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 --2- 172.21.15.105:0/112573143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb69c101080 0x7fb69c1974a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 >> 172.21.15.105:0/112573143 conn(0x7fb69c078000 msgr2=0x7fb69c0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 shutdown_connections 2024-04-22T06:38:49.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:49.535+0000 7fb6a0be2640 1 -- 172.21.15.105:0/112573143 wait complete. 2024-04-22T06:38:49.546 INFO:tasks.cephfs.filesystem:scrub status for tag:1dc9ba3f-9cb9-4d1d-a95d-73a068cfc70f - {'path': '/', 'tag': '1dc9ba3f-9cb9-4d1d-a95d-73a068cfc70f', 'options': 'recursive,force'} 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 977 B/s wr, 38 op/s 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: scrub queued for path: / 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: scrub summary: active paths [/] 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:50 smithi177 ceph-mon[31346]: scrub summary: idle 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 977 B/s wr, 38 op/s 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: scrub queued for path: / 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: scrub summary: active paths [/] 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.819 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:50 smithi105 ceph-mon[27916]: scrub summary: idle 2024-04-22T06:38:50.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 977 B/s wr, 38 op/s 2024-04-22T06:38:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: scrub queued for path: / 2024-04-22T06:38:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: scrub summary: active paths [/] 2024-04-22T06:38:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: scrub summary: idle+waiting paths [/] 2024-04-22T06:38:50.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:50 smithi112 ceph-mon[31236]: scrub summary: idle 2024-04-22T06:38:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 -- 172.21.15.105:0/3829757777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c101aa0 msgr2=0x7f3d9c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 --2- 172.21.15.105:0/3829757777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c101aa0 0x7f3d9c105a90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c009900 tx=0x7f3d8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 -- 172.21.15.105:0/3829757777 shutdown_connections 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 --2- 172.21.15.105:0/3829757777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 --2- 172.21.15.105:0/3829757777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c101aa0 0x7f3d9c105a90 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 --2- 172.21.15.105:0/3829757777 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.219+0000 7f3da4255640 1 -- 172.21.15.105:0/3829757777 >> 172.21.15.105:0/3829757777 conn(0x7f3d9c077fd0 msgr2=0x7f3d9c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.220+0000 7f3da4255640 1 -- 172.21.15.105:0/3829757777 shutdown_connections 2024-04-22T06:38:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.220+0000 7f3da4255640 1 -- 172.21.15.105:0/3829757777 wait complete. 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.220+0000 7f3da4255640 1 Processor -- start 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.220+0000 7f3da4255640 1 -- start start 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1110e0 con 0x7f3d9c19be50 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57734/0 (socket says 172.21.15.105:57734) 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 -- 172.21.15.105:0/1615344171 learned_addr learned my addr 172.21.15.105:0/1615344171 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c110f60 con 0x7f3d9c106170 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c111260 con 0x7f3d9c1010f0 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da17f0640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 -- 172.21.15.105:0/1615344171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 msgr2=0x7f3d9c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 -- 172.21.15.105:0/1615344171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 msgr2=0x7f3d9c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 -- 172.21.15.105:0/1615344171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c19c940 con 0x7f3d9c19be50 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da17f0640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.221+0000 7f3da27f2640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400eb10 tx=0x7f3d9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.222+0000 7f3d92ffd640 1 -- 172.21.15.105:0/1615344171 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94004b20 con 0x7f3d9c19be50 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.222+0000 7f3d92ffd640 1 -- 172.21.15.105:0/1615344171 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d94004cc0 con 0x7f3d9c19be50 2024-04-22T06:38:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.222+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c19cc30 con 0x7f3d9c19be50 2024-04-22T06:38:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.222+0000 7f3d92ffd640 1 -- 172.21.15.105:0/1615344171 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94010670 con 0x7f3d9c19be50 2024-04-22T06:38:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.222+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c1a43a0 con 0x7f3d9c19be50 2024-04-22T06:38:51.223 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.225+0000 7f3d92ffd640 1 -- 172.21.15.105:0/1615344171 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f3d94005000 con 0x7f3d9c19be50 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 Processor -- start 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3d92ffd640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 0x7f3d880a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 -- start start 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3d92ffd640 1 -- 172.21.15.105:0/1615344171 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3d940d11e0 con 0x7f3d9c19be50 2024-04-22T06:38:51.225 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7c620 0x55c395f7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da17f0640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 0x7f3d880a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7cfe0 0x55c395f90ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c395ee18d0 con 0x55c395f7c620 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c395e7ee80 con 0x55c395f7cfe0 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.226+0000 7f3da6e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c395edd890 con 0x55c395f7d8d0 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53500/0 (socket says 172.21.15.105:53500) 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 -- 172.21.15.105:0/1797556441 learned_addr learned my addr 172.21.15.105:0/1797556441 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da27f2640 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7cfe0 0x55c395f90ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 -- 172.21.15.105:0/1797556441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7cfe0 msgr2=0x55c395f90ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7cfe0 0x55c395f90ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 -- 172.21.15.105:0/1797556441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7c620 msgr2=0x55c395f7caa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-22T06:38:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7c620 0x55c395f7caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 -- 172.21.15.105:0/1797556441 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c395ea0320 con 0x55c395f7d8d0 2024-04-22T06:38:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da17f0640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 0x7f3d880a6310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02c240 tx=0x7f3d8c005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.227+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d98011540 tx=0x7f3d98011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40684b45fcab9b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.228+0000 7f3d87fff640 1 -- 172.21.15.105:0/1797556441 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800f640 con 0x55c395f7d8d0 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.228+0000 7f3d87fff640 1 -- 172.21.15.105:0/1797556441 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d9800fc90 con 0x55c395f7d8d0 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.228+0000 7f3d87fff640 1 -- 172.21.15.105:0/1797556441 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98019cc0 con 0x55c395f7d8d0 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- 172.21.15.105:0/1797556441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 msgr2=0x55c395f840d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d98011540 tx=0x7f3d98011a00 comp rx=0 tx=0).stop 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- 172.21.15.105:0/1797556441 shutdown_connections 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7cfe0 0x55c395f90ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7c620 0x55c395f7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 --2- 172.21.15.105:0/1797556441 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7d8d0 0x55c395f840d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- 172.21.15.105:0/1797556441 >> 172.21.15.105:0/1797556441 conn(0x55c395f43c90 msgr2=0x55c395f77850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- 172.21.15.105:0/1797556441 shutdown_connections 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- 172.21.15.105:0/1797556441 wait complete. 2024-04-22T06:38:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 -- start start 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.229+0000 7f3da6e46740 1 Processor -- start 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f83f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f812f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f81850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c395f91650 con 0x55c395f7d8d0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c395e69eb0 con 0x55c395f7c620 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55c395f917d0 con 0x55c395f7cfe0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da6e46740 10 client.0 _ll_register_callbacks cb 0x55c395e96b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da27f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f81850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da17f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f812f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f83f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da17f0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f812f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53502/0 (socket says 172.21.15.105:53502) 2024-04-22T06:38:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da27f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f81850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57740/0 (socket says 172.21.15.105:57740) 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da17f0640 1 -- 172.21.15.105:0/546215894 learned_addr learned my addr 172.21.15.105:0/546215894 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 -- 172.21.15.105:0/546215894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 msgr2=0x55c395f812f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f812f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 -- 172.21.15.105:0/546215894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 msgr2=0x55c395f81850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f81850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.230+0000 7f3da1ff1640 1 -- 172.21.15.105:0/546215894 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c395bf4210 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f83f60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d9800fbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55cf64d1ee8e069 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3d857fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800fe40 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3d98019ad0 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d9800fc90 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3d98030050 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3da6e46740 10 client.25574 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c395cc0620 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.231+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d9800fc90 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.232+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f3d98005040 con 0x55c395f7c620 2024-04-22T06:38:51.234 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.232+0000 7f3da6e46740 10 client.25574 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.232+0000 7f3da6e46740 10 client.25574 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da6e46740 20 client.25574 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c395fa0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da6e46740 4 client.25574 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c395ef5880 con 0x55c395f9e620 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da17f0640 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c395fa0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3da17f0640 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c395fa0a10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a47d23039519e12 server_cookie=80421010615323ba in_seq=0 out_seq=0 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.233+0000 7f3d857fa640 10 client.25574 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3d857fa640 1 -- 172.21.15.105:0/546215894 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c395ef5880 con 0x55c395f9e620 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3d857fa640 10 client.25574 handle_command_reply: tid=0 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 client.25574 shutdown 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 msgr2=0x55c395f83f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f83f60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3d98000c00 tx=0x7f3d9800fbe0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 shutdown_connections 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c395fa0a10 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f81850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3d857fa640 0 client.25574 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f812f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 --2- 172.21.15.105:0/546215894 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f83f60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.234+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 >> 172.21.15.105:0/546215894 conn(0x55c395f43c90 msgr2=0x55c395f795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.235+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 shutdown_connections 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.235+0000 7f3da6e46740 1 -- 172.21.15.105:0/546215894 wait complete. 2024-04-22T06:38:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.235+0000 7f3da6e46740 20 client.25574 trim_cache size 0 max 16384 2024-04-22T06:38:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.263+0000 7f3da6e46740 1 Processor -- start 2024-04-22T06:38:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.263+0000 7f3da6e46740 1 -- start start 2024-04-22T06:38:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f83a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f83f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1ab760 con 0x55c395f7d8d0 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1ad2a0 con 0x55c395f7c620 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da6e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1abe30 con 0x55c395f7cfe0 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da1ff1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da1ff1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:51542/0 (socket says 172.21.15.105:51542) 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da1ff1640 1 -- 172.21.15.105:0/563198145 learned_addr learned my addr 172.21.15.105:0/563198145 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da17f0640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f83a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da27f2640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f83f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.264+0000 7f3da1ff1640 1 -- 172.21.15.105:0/563198145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 msgr2=0x55c395f83a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f83a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da1ff1640 1 -- 172.21.15.105:0/563198145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 msgr2=0x55c395f83f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f83f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da1ff1640 1 -- 172.21.15.105:0/563198145 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c395f72f30 con 0x55c395f7c620 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da27f2640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f83f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d980058c0 tx=0x7f3d9800f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5c0aec82359f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3d877fe640 1 -- 172.21.15.105:0/563198145 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980050d0 con 0x55c395f7c620 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3d877fe640 1 -- 172.21.15.105:0/563198145 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d98027070 con 0x55c395f7c620 2024-04-22T06:38:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.265+0000 7f3d877fe640 1 -- 172.21.15.105:0/563198145 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98003920 con 0x55c395f7c620 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 -- 172.21.15.105:0/563198145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 msgr2=0x55c395f9a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d980058c0 tx=0x7f3d9800f8d0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 -- 172.21.15.105:0/563198145 shutdown_connections 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7d8d0 0x55c395f83f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7cfe0 0x55c395f83a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 --2- 172.21.15.105:0/563198145 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7c620 0x55c395f9a550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 -- 172.21.15.105:0/563198145 >> 172.21.15.105:0/563198145 conn(0x55c395f43c90 msgr2=0x55c395f90910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 -- 172.21.15.105:0/563198145 shutdown_connections 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.266+0000 7f3da6e46740 1 -- 172.21.15.105:0/563198145 wait complete. 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.267+0000 7f3da6e46740 1 -- start start 2024-04-22T06:38:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.267+0000 7f3da6e46740 1 Processor -- start 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.267+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7c620 0x55c395f96a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.267+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.267+0000 7f3da6e46740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 0x55c395f82970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da6e46740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1acbd0 con 0x55c395f7cfe0 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da6e46740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1ac500 con 0x55c395f7d8d0 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57758/0 (socket says 172.21.15.105:57758) 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 -- 172.21.15.105:0/3783380498 learned_addr learned my addr 172.21.15.105:0/3783380498 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da6e46740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c1a9740 con 0x55c395f7c620 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7c620 0x55c395f96a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da6e46740 10 client.0 _ll_register_callbacks cb 0x7f3d9400cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 -- 172.21.15.105:0/3783380498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7c620 msgr2=0x55c395f96a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7c620 0x55c395f96a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da27f2640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 0x55c395f82970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 -- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 msgr2=0x55c395f82970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 0x55c395f82970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 -- 172.21.15.105:0/3783380498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c395f82fb0 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da27f2640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 0x55c395f82970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3da17f0640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c036040 tx=0x7f3d8c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7668338b8c810a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3d857fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.268+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c005450 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3d8c0055f0 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d98027070 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3d8c0046c0 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 10 client.16272 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c396004280 con 0x55c395f7cfe0 2024-04-22T06:38:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d8c004d10 con 0x55c395f7cfe0 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f3d8c057070 con 0x55c395f7cfe0 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 10 client.16272 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 20 client.16272 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c396006a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 4 client.16272 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3d98027070 con 0x55c395f9e620 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.269+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c396006a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.270+0000 7f3d857fa640 10 client.16272 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.270+0000 7f3da1ff1640 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c396006a70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61a73440ca13c88e server_cookie=c9a6d235a7cd834c in_seq=0 out_seq=0 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3d857fa640 1 -- 172.21.15.105:0/3783380498 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3d98027070 con 0x55c395f9e620 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3d857fa640 10 client.16272 handle_command_reply: tid=0 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 client.16272 shutdown 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 msgr2=0x55c395f96f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c036040 tx=0x7f3d8c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 shutdown_connections 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55c395f9e620 0x55c396006a70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c395f7d8d0 0x55c395f82970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c395f7cfe0 0x55c395f96f70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3d857fa640 0 client.16272 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 --2- 172.21.15.105:0/3783380498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55c395f7c620 0x55c395f96a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 >> 172.21.15.105:0/3783380498 conn(0x55c395f43c90 msgr2=0x55c395f76040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 shutdown_connections 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 1 -- 172.21.15.105:0/3783380498 wait complete. 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.274+0000 7f3da6e46740 20 client.16272 trim_cache size 0 max 16384 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:38:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 msgr2=0x7f3d880a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 0x7f3d880a6310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c02c240 tx=0x7f3d8c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 msgr2=0x7f3d9c19c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400eb10 tx=0x7f3d9400c2d0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 shutdown_connections 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.275+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3d880a3e50 0x7f3d880a6310 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d9c19be50 0x7f3d9c19c300 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d9c106170 0x7f3d9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 --2- 172.21.15.105:0/1615344171 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d9c1010f0 0x7f3d9c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 >> 172.21.15.105:0/1615344171 conn(0x7f3d9c077fd0 msgr2=0x7f3d9c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 shutdown_connections 2024-04-22T06:38:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:38:51.276+0000 7f3da4255640 1 -- 172.21.15.105:0/1615344171 wait complete. 2024-04-22T06:38:52.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:52 smithi177 ceph-mon[31346]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 938 B/s wr, 31 op/s 2024-04-22T06:38:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:52 smithi105 ceph-mon[27916]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 938 B/s wr, 31 op/s 2024-04-22T06:38:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:52 smithi112 ceph-mon[31236]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 938 B/s wr, 31 op/s 2024-04-22T06:38:54.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:54 smithi177 ceph-mon[31346]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 KiB/s wr, 36 op/s 2024-04-22T06:38:54.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:54 smithi105 ceph-mon[27916]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 KiB/s wr, 36 op/s 2024-04-22T06:38:54.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:54 smithi112 ceph-mon[31236]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 KiB/s wr, 36 op/s 2024-04-22T06:38:55.823 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.5 2024-04-22T06:38:55.824 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:38:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 -- 172.21.15.105:0/3527449397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c1011a0 msgr2=0x7fab2c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 --2- 172.21.15.105:0/3527449397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c1011a0 0x7fab2c101620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0099b0 tx=0x7fab1c02de60 comp rx=0 tx=0).stop 2024-04-22T06:38:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 -- 172.21.15.105:0/3527449397 shutdown_connections 2024-04-22T06:38:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 --2- 172.21.15.105:0/3527449397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c101b60 0x7fab2c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 --2- 172.21.15.105:0/3527449397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c1011a0 0x7fab2c101620 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 --2- 172.21.15.105:0/3527449397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.985+0000 7fab336c7640 1 -- 172.21.15.105:0/3527449397 >> 172.21.15.105:0/3527449397 conn(0x7fab2c077e30 msgr2=0x7fab2c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.986+0000 7fab336c7640 1 -- 172.21.15.105:0/3527449397 shutdown_connections 2024-04-22T06:38:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.986+0000 7fab336c7640 1 -- 172.21.15.105:0/3527449397 wait complete. 2024-04-22T06:38:55.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.986+0000 7fab336c7640 1 Processor -- start 2024-04-22T06:38:55.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.986+0000 7fab336c7640 1 -- start start 2024-04-22T06:38:55.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 0x7fab2c196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:55.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:55.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab30c62640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab30c62640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38370/0 (socket says 172.21.15.105:38370) 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab30c62640 1 -- 172.21.15.105:0/3989215917 learned_addr learned my addr 172.21.15.105:0/3989215917 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c110020 con 0x7fab2c101b60 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab31463640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 0x7fab2c196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c10fea0 con 0x7fab2c1011a0 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1101a0 con 0x7fab2c105e30 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.987+0000 7fab31c64640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 -- 172.21.15.105:0/3989215917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 msgr2=0x7fab2c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 -- 172.21.15.105:0/3989215917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 msgr2=0x7fab2c196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 0x7fab2c196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c19b550 con 0x7fab2c101b60 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab31463640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 0x7fab2c196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:55.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab30c62640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009230 tx=0x7fab1c004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.988+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c044020 con 0x7fab2c101b60 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.989+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1c040460 con 0x7fab2c101b60 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.989+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c19b7e0 con 0x7fab2c101b60 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.989+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0366a0 con 0x7fab2c101b60 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.989+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1a30d0 con 0x7fab2c101b60 2024-04-22T06:38:55.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.990+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c068de0 con 0x7fab2c101b60 2024-04-22T06:38:55.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.992+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fab1c047020 con 0x7fab2c101b60 2024-04-22T06:38:55.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.995+0000 7fab227fc640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 0x7fab180a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:55.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.996+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fab1c0f4b10 con 0x7fab2c101b60 2024-04-22T06:38:55.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.996+0000 7fab31463640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 0x7fab180a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:55.995 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:55.997+0000 7fab31463640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 0x7fab180a64e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab2800b390 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.000 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.002+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab1c04e020 con 0x7fab2c101b60 2024-04-22T06:38:56.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.450+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab2c109330 con 0x7fab2c101b60 2024-04-22T06:38:56.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.451+0000 7fab227fc640 1 -- 172.21.15.105:0/3989215917 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11943 (secure 0 0 0) 0x7fab1c0c1020 con 0x7fab2c101b60 2024-04-22T06:38:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:38:56.450 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:38:41.031516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24431,"mds_2":24398,"mds_3":24397,"mds_4":14622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":31,"state":"up:active","state_seq":51,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":33,"state":"up:active","state_seq":55,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24397,14622,24398,24500,24431]},"id":1}]} 2024-04-22T06:38:56.450 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 37 2024-04-22T06:38:56.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 msgr2=0x7fab180a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 0x7fab180a64e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab2800b390 comp rx=0 tx=0).stop 2024-04-22T06:38:56.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 msgr2=0x7fab2c196aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009230 tx=0x7fab1c004870 comp rx=0 tx=0).stop 2024-04-22T06:38:56.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 shutdown_connections 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fab180a4020 0x7fab180a64e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fab2c105e30 0x7fab2c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fab2c101b60 0x7fab2c196aa0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 --2- 172.21.15.105:0/3989215917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab2c1011a0 0x7fab2c196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.454+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 >> 172.21.15.105:0/3989215917 conn(0x7fab2c077e30 msgr2=0x7fab2c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.455+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 shutdown_connections 2024-04-22T06:38:56.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.455+0000 7fab336c7640 1 -- 172.21.15.105:0/3989215917 wait complete. 2024-04-22T06:38:56.468 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 60 --expiration 77.5 --await' 2024-04-22T06:38:56.468 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db / --timeout 60 --expiration 77.5 --await 2024-04-22T06:38:56.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- 172.21.15.105:0/1499262950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14074320 msgr2=0x7fae14074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 --2- 172.21.15.105:0/1499262950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14074320 0x7fae14074700 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fae04009900 tx=0x7fae0402bcd0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- 172.21.15.105:0/1499262950 shutdown_connections 2024-04-22T06:38:56.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 --2- 172.21.15.105:0/1499262950 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14110be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 --2- 172.21.15.105:0/1499262950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 0x7fae1410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 --2- 172.21.15.105:0/1499262950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14074320 0x7fae14074700 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- 172.21.15.105:0/1499262950 >> 172.21.15.105:0/1499262950 conn(0x7fae140fd9e0 msgr2=0x7fae140ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- 172.21.15.105:0/1499262950 shutdown_connections 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- 172.21.15.105:0/1499262950 wait complete. 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 Processor -- start 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.585+0000 7fae1a281640 1 -- start start 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 0x7fae14101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 0x7fae141ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae14114400 con 0x7fae14102970 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae14114280 con 0x7fae14074c40 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1a281640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae14114580 con 0x7fae1410e7d0 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 0x7fae141ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53170/0 (socket says 172.21.15.105:53170) 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 0x7fae141ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38384/0 (socket says 172.21.15.105:38384) 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 -- 172.21.15.105:0/1838662749 learned_addr learned my addr 172.21.15.105:0/1838662749 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae13fff640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 0x7fae14101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 -- 172.21.15.105:0/1838662749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 msgr2=0x7fae14101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 0x7fae14101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 -- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 msgr2=0x7fae141ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 0x7fae141ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.586+0000 7fae137fe640 1 -- 172.21.15.105:0/1838662749 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae141ab6d0 con 0x7fae1410e7d0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae137fe640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9f0 tx=0x7fae0800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae117fa640 1 -- 172.21.15.105:0/1838662749 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0800cd80 con 0x7fae1410e7d0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae141ab9c0 con 0x7fae1410e7d0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae117fa640 1 -- 172.21.15.105:0/1838662749 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae080108e0 con 0x7fae1410e7d0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae117fa640 1 -- 172.21.15.105:0/1838662749 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae08018980 con 0x7fae1410e7d0 2024-04-22T06:38:56.590 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.587+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae141abf30 con 0x7fae1410e7d0 2024-04-22T06:38:56.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.590+0000 7fae1ce73740 1 Processor -- start 2024-04-22T06:38:56.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.590+0000 7fae1ce73740 1 -- start start 2024-04-22T06:38:56.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.590+0000 7fae117fa640 1 -- 172.21.15.105:0/1838662749 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7fae08010430 con 0x7fae1410e7d0 2024-04-22T06:38:56.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.590+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3572da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad357bec0 0x55aad357e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1ce73740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aad34d30b0 con 0x55aad35732e0 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1ce73740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aad3484680 con 0x55aad357bec0 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1ce73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55aad3490c60 con 0x55aad35729c0 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38386/0 (socket says 172.21.15.105:38386) 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 -- 172.21.15.105:0/3934141443 learned_addr learned my addr 172.21.15.105:0/3934141443 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae1881e640 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3572da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae137fe640 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad357bec0 0x55aad357e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 -- 172.21.15.105:0/3934141443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 msgr2=0x55aad3572da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3572da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 -- 172.21.15.105:0/3934141443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad357bec0 msgr2=0x55aad357e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad357bec0 0x55aad357e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 -- 172.21.15.105:0/3934141443 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aad32b5c90 con 0x55aad35732e0 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae117fa640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 0x7fadfc0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.591+0000 7fae13fff640 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae04009900 tx=0x7fae04005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7b62c0f336c37d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.592+0000 7fae1881e640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 0x7fadfc0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.592+0000 7fae117fa640 1 -- 172.21.15.105:0/1838662749 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fae080d0e40 con 0x7fae1410e7d0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.592+0000 7fadfa7fc640 1 -- 172.21.15.105:0/3934141443 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04004040 con 0x55aad35732e0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.592+0000 7fadfa7fc640 1 -- 172.21.15.105:0/3934141443 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae040054f0 con 0x55aad35732e0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fadfa7fc640 1 -- 172.21.15.105:0/3934141443 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0402ccd0 con 0x55aad35732e0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- 172.21.15.105:0/3934141443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 msgr2=0x55aad357b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae04009900 tx=0x7fae04005fd0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1881e640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 0x7fadfc0a6410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae1406b630 tx=0x7fae0c009500 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- 172.21.15.105:0/3934141443 shutdown_connections 2024-04-22T06:38:56.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad357bec0 0x55aad357e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35732e0 0x55aad357b980 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3934141443 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3572da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- 172.21.15.105:0/3934141443 >> 172.21.15.105:0/3934141443 conn(0x55aad356d0f0 msgr2=0x55aad356d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- 172.21.15.105:0/3934141443 shutdown_connections 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- 172.21.15.105:0/3934141443 wait complete. 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 -- start start 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.593+0000 7fae1ce73740 1 Processor -- start 2024-04-22T06:38:56.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3578e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad3579350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55aad34d6400 con 0x55aad357bec0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aad3492840 con 0x55aad35732e0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55aad34db720 con 0x55aad35729c0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1ce73740 10 client.0 _ll_register_callbacks cb 0x55aad34e3ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae137fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad3579350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38388/0 (socket says 172.21.15.105:38388) 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 -- 172.21.15.105:0/3036586026 learned_addr learned my addr 172.21.15.105:0/3036586026 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae13fff640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3578e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 -- 172.21.15.105:0/3036586026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 msgr2=0x55aad3578e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3578e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 -- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 msgr2=0x55aad3579350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad3579350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.594+0000 7fae1881e640 1 -- 172.21.15.105:0/3036586026 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aad3316420 con 0x55aad357bec0 2024-04-22T06:38:56.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fae1881e640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fae0c012010 tx=0x7fae0c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773d6d6466bdcf1b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fadf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c004b20 con 0x55aad357bec0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fae0c004cc0 con 0x55aad357bec0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae040054f0 con 0x55aad357bec0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fae0c00f890 con 0x55aad357bec0 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fae1ce73740 10 client.16290 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:56.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55aad31a9580 con 0x55aad357bec0 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.595+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae040054f0 con 0x55aad357bec0 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fae0c026070 con 0x55aad357bec0 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 10 client.16290 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 10 client.16290 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 20 client.16290 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad358bc40 0x55aad358e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 4 client.16290 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:38:56.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aad34e4b10 con 0x55aad358bc40 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae13fff640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad358bc40 0x55aad358e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fadf37fe640 10 client.16290 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.596+0000 7fae13fff640 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad358bc40 0x55aad358e050 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0bf6eed0d1b96c server_cookie=93d77c0d80c72798 in_seq=0 out_seq=0 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fadf37fe640 1 -- 172.21.15.105:0/3036586026 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aad34e4b10 con 0x55aad358bc40 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fadf37fe640 10 client.16290 handle_command_reply: tid=0 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fae1ce73740 1 client.16290 shutdown 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 msgr2=0x55aad3579890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.607 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fae0c012010 tx=0x7fae0c004ab0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 shutdown_connections 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.597+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad358bc40 0x55aad358e050 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad3579890 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad3579350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 --2- 172.21.15.105:0/3036586026 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35729c0 0x55aad3578e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 >> 172.21.15.105:0/3036586026 conn(0x55aad356d0f0 msgr2=0x55aad356ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fadf37fe640 0 client.16290 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:56.608 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 shutdown_connections 2024-04-22T06:38:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 1 -- 172.21.15.105:0/3036586026 wait complete. 2024-04-22T06:38:56.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.598+0000 7fae1ce73740 20 client.16290 trim_cache size 0 max 16384 2024-04-22T06:38:56.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.631+0000 7fae1ce73740 1 Processor -- start 2024-04-22T06:38:56.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 -- start start 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 0x55aad3587890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35732e0 0x55aad3577f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b63e0 con 0x55aad357bec0 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b4f70 con 0x55aad35729c0 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1ce73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b5640 con 0x55aad35732e0 2024-04-22T06:38:56.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38396/0 (socket says 172.21.15.105:38396) 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 -- 172.21.15.105:0/460819308 learned_addr learned my addr 172.21.15.105:0/460819308 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae137fe640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35732e0 0x55aad3577f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae13fff640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 0x55aad3587890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 -- 172.21.15.105:0/460819308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35732e0 msgr2=0x55aad3577f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35732e0 0x55aad3577f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.638 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 -- 172.21.15.105:0/460819308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 msgr2=0x55aad3587890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 0x55aad3587890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae1881e640 1 -- 172.21.15.105:0/460819308 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aad356cf20 con 0x55aad357bec0 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.632+0000 7fae13fff640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 0x55aad3587890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fae1881e640 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fae0c004010 tx=0x7fae0c011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ad01156b90faa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fadf9ffb640 1 -- 172.21.15.105:0/460819308 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c003d70 con 0x55aad357bec0 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fadf9ffb640 1 -- 172.21.15.105:0/460819308 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae0c01e750 con 0x55aad357bec0 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fadf9ffb640 1 -- 172.21.15.105:0/460819308 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c02d660 con 0x55aad357bec0 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fae1ce73740 1 -- 172.21.15.105:0/460819308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 msgr2=0x55aad357ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.639 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.633+0000 7fae1ce73740 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fae0c004010 tx=0x7fae0c011440 comp rx=0 tx=0).stop 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- 172.21.15.105:0/460819308 shutdown_connections 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad357bec0 0x55aad357ac40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad35732e0 0x55aad3577f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- 172.21.15.105:0/460819308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35729c0 0x55aad3587890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- 172.21.15.105:0/460819308 >> 172.21.15.105:0/460819308 conn(0x55aad356d0f0 msgr2=0x55aad357a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- 172.21.15.105:0/460819308 shutdown_connections 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- 172.21.15.105:0/460819308 wait complete. 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- start start 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 Processor -- start 2024-04-22T06:38:56.640 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 0x55aad357fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad357ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b2880 con 0x55aad35729c0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b5d10 con 0x55aad35732e0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b2ef0 con 0x55aad357bec0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.634+0000 7fae1ce73740 10 client.0 _ll_register_callbacks cb 0x7fae0c009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae13fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 0x55aad357fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.641 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:53214/0 (socket says 172.21.15.105:53214) 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 -- 172.21.15.105:0/394753907 learned_addr learned my addr 172.21.15.105:0/394753907 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae137fe640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad357ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 -- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 msgr2=0x55aad357ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad357ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 -- 172.21.15.105:0/394753907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 msgr2=0x55aad357fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:38:56.642 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 0x55aad357fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 -- 172.21.15.105:0/394753907 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aad356c950 con 0x55aad357bec0 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae137fe640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad357ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae13fff640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 0x55aad357fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1881e640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae0c018040 tx=0x7fae0c005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=542ecbb247c4960c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fadf37fe640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c022040 con 0x55aad357bec0 2024-04-22T06:38:56.643 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae0c01e750 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.635+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fae0c01ed10 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fae0c01c400 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 10 client.25618 fetch_fsmap learned FSMap version 37 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55aad358a740 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae0c01e750 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fae0c026070 con 0x55aad357bec0 2024-04-22T06:38:56.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 10 client.25618 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 10 client.25618 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 20 client.25618 populate_metadata read hostname 'smithi105' 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad35f2a10 0x55aad35f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 4 client.25618 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.5, "await": true}] 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.636+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.5, "await": true}) -- 0x55aad34de980 con 0x55aad35f2a10 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.637+0000 7fae13fff640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad35f2a10 0x55aad35f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:38:56.645 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.637+0000 7fadf37fe640 10 client.25618 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:56.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.637+0000 7fae13fff640 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad35f2a10 0x55aad35f4e00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d8364a3e4a374f server_cookie=98e059297e7c6ada in_seq=0 out_seq=0 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fadf37fe640 1 -- 172.21.15.105:0/394753907 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55aad34de980 con 0x55aad35f2a10 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fadf37fe640 10 client.25618 handle_command_reply: tid=0 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 client.25618 shutdown 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 msgr2=0x55aad3580520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fae0c018040 tx=0x7fae0c005be0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 shutdown_connections 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55aad35f2a10 0x55aad35f4e00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55aad357bec0 0x55aad3580520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fadf37fe640 0 client.25618 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aad35732e0 0x55aad357ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 --2- 172.21.15.105:0/394753907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55aad35729c0 0x55aad357fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.645+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 >> 172.21.15.105:0/394753907 conn(0x55aad356d0f0 msgr2=0x55aad356b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 shutdown_connections 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1ce73740 1 -- 172.21.15.105:0/394753907 wait complete. 2024-04-22T06:38:56.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1ce73740 20 client.25618 trim_cache size 0 max 16384 2024-04-22T06:38:56.650 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:38:56.650 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 37, 2024-04-22T06:38:56.650 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 6, 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "824d2692": { 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 6, 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:56.651 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 77.5, 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:38:56.652 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:56.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:56.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:56.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:38:56.654 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:38:56.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 msgr2=0x7fadfc0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 0x7fadfc0a6410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae1406b630 tx=0x7fae0c009500 comp rx=0 tx=0).stop 2024-04-22T06:38:56.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 msgr2=0x7fae14102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:38:56.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae0800e9f0 tx=0x7fae0800eeb0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 shutdown_connections 2024-04-22T06:38:56.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fadfc0a3f50 0x7fadfc0a6410 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae14102970 0x7fae141ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fae1410e7d0 0x7fae14102430 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 --2- 172.21.15.105:0/1838662749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fae14074c40 0x7fae14101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:38:56.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 >> 172.21.15.105:0/1838662749 conn(0x7fae140fd9e0 msgr2=0x7fae140ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:38:56.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 shutdown_connections 2024-04-22T06:38:56.657 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:38:56.646+0000 7fae1a281640 1 -- 172.21.15.105:0/1838662749 wait complete. 2024-04-22T06:38:56.659 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 824d2692, quiesce took 0.8 seconds. Will release after: 17.5 2024-04-22T06:38:56.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:56 smithi177 ceph-mon[31346]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 426 B/s wr, 13 op/s 2024-04-22T06:38:56.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:56 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3989215917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:56.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:56 smithi105 ceph-mon[27916]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 426 B/s wr, 13 op/s 2024-04-22T06:38:56.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:56 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3989215917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:56 smithi112 ceph-mon[31236]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 426 B/s wr, 13 op/s 2024-04-22T06:38:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:56 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3989215917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:38:58.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:38:58 smithi177 ceph-mon[31346]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:38:58.760 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:38:58 smithi112 ceph-mon[31236]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:38:58.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:38:58 smithi105 ceph-mon[27916]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:39:00.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:00 smithi177 ceph-mon[31346]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-04-22T06:39:00.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:00 smithi105 ceph-mon[27916]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-04-22T06:39:00.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:00 smithi112 ceph-mon[31236]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-04-22T06:39:02.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:02 smithi177 ceph-mon[31346]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:39:02.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:02 smithi105 ceph-mon[27916]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:39:02.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:02 smithi112 ceph-mon[31236]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-22T06:39:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 -- 172.21.15.105:0/985531761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78101aa0 msgr2=0x7f9f78105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/985531761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78101aa0 0x7f9f78105a90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009900 tx=0x7f9f6402bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 -- 172.21.15.105:0/985531761 shutdown_connections 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/985531761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f78106170 0x7f9f7810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/985531761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78101aa0 0x7f9f78105a90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/985531761 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 0x7f9f781014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 -- 172.21.15.105:0/985531761 >> 172.21.15.105:0/985531761 conn(0x7f9f78077fd0 msgr2=0x7f9f780783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 -- 172.21.15.105:0/985531761 shutdown_connections 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.414+0000 7f9f7ed75640 1 -- 172.21.15.105:0/985531761 wait complete. 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 Processor -- start 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 -- start start 2024-04-22T06:39:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 0x7f9f7819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781110e0 con 0x7f9f78106170 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38438/0 (socket says 172.21.15.105:38438) 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f77fff640 1 -- 172.21.15.105:0/1340858180 learned_addr learned my addr 172.21.15.105:0/1340858180 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 0x7f9f7819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78110f60 con 0x7f9f781975f0 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.415+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78111260 con 0x7f9f781010f0 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 -- 172.21.15.105:0/1340858180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 msgr2=0x7f9f7819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 0x7f9f7819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 -- 172.21.15.105:0/1340858180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 msgr2=0x7f9f78197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 -- 172.21.15.105:0/1340858180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f78198110 con 0x7f9f78106170 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9f6402c240 tx=0x7f9f6402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f75ffb640 1 -- 172.21.15.105:0/1340858180 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64043020 con 0x7f9f78106170 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f75ffb640 1 -- 172.21.15.105:0/1340858180 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f6402cbe0 con 0x7f9f78106170 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f75ffb640 1 -- 172.21.15.105:0/1340858180 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64003b20 con 0x7f9f78106170 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.416+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f781983a0 con 0x7f9f78106170 2024-04-22T06:39:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.417+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f780697e0 con 0x7f9f78106170 2024-04-22T06:39:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.420+0000 7f9f75ffb640 1 -- 172.21.15.105:0/1340858180 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138931+0+0 (secure 0 0 0) 0x7f9f64046020 con 0x7f9f78106170 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.420+0000 7f9f815fd740 1 Processor -- start 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.420+0000 7f9f815fd740 1 -- start start 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a641afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a64264d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586a637f8d0 con 0x5586a641b8d0 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586a631ce80 con 0x5586a641ab60 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f815fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586a637b890 con 0x5586a64240e0 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f75ffb640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 0x7f9f680a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38450/0 (socket says 172.21.15.105:38450) 2024-04-22T06:39:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 -- 172.21.15.105:0/3707335544 learned_addr learned my addr 172.21.15.105:0/3707335544 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a641afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 0x7f9f680a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f77fff640 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a64264d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 -- 172.21.15.105:0/3707335544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 msgr2=0x5586a64264d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f75ffb640 1 -- 172.21.15.105:0/1340858180 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f9f640f2050 con 0x7f9f78106170 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a64264d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 -- 172.21.15.105:0/3707335544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 msgr2=0x5586a641afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a641afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.421+0000 7f9f7d312640 1 -- 172.21.15.105:0/3707335544 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586a633e320 con 0x5586a641b8d0 2024-04-22T06:39:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.422+0000 7f9f7d312640 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9f70012d60 tx=0x7f9f70010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86abc5fed4364b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.422+0000 7f9f62ffd640 1 -- 172.21.15.105:0/3707335544 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7001d7e0 con 0x5586a641b8d0 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.422+0000 7f9f62ffd640 1 -- 172.21.15.105:0/3707335544 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7001de30 con 0x5586a641b8d0 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.422+0000 7f9f62ffd640 1 -- 172.21.15.105:0/3707335544 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7001cd30 con 0x5586a641b8d0 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 0x7f9f680a6430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f7001cfd0 tx=0x7f9f7000e3a0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 -- 172.21.15.105:0/3707335544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 msgr2=0x5586a641a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9f70012d60 tx=0x7f9f70010740 comp rx=0 tx=0).stop 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 -- 172.21.15.105:0/3707335544 shutdown_connections 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a64264d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a641afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3707335544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a641a620 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 -- 172.21.15.105:0/3707335544 >> 172.21.15.105:0/3707335544 conn(0x5586a63e1c90 msgr2=0x5586a6415850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 -- 172.21.15.105:0/3707335544 shutdown_connections 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.423+0000 7f9f815fd740 1 -- 172.21.15.105:0/3707335544 wait complete. 2024-04-22T06:39:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 -- start start 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 Processor -- start 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a6423a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a6421340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5586a6426e10 con 0x5586a641b8d0 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.424+0000 7f9f815fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5586a6307eb0 con 0x5586a641ab60 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f815fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x5586a6426f90 con 0x5586a64240e0 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f815fd740 10 client.0 _ll_register_callbacks cb 0x5586a6334b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38458/0 (socket says 172.21.15.105:38458) 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 -- 172.21.15.105:0/3550146024 learned_addr learned my addr 172.21.15.105:0/3550146024 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a6423a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f7d312640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a6421340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 -- 172.21.15.105:0/3550146024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 msgr2=0x5586a6421340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a6421340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 -- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 msgr2=0x5586a6423a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a6423a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 -- 172.21.15.105:0/3550146024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586a6092210 con 0x5586a641b8d0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f7d312640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a6421340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a6423a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f77fff640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ef670 tx=0x7f9f640bd8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0e14bb90b1006f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.425+0000 7f9f57fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f640c1030 con 0x5586a641b8d0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f9f64003cc0 con 0x5586a641b8d0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f7001de30 con 0x5586a641b8d0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9f640c6a40 con 0x5586a641b8d0 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 10 client.16308 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5586a615e620 con 0x5586a641b8d0 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7001de30 con 0x5586a641b8d0 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f9f640c6640 con 0x5586a641b8d0 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 10 client.16308 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 20 client.16308 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a64340f0 0x5586a6436540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 4 client.16308 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.426+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586a6393880 con 0x5586a64340f0 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.427+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a64340f0 0x5586a6436540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.427+0000 7f9f57fff640 10 client.16308 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.427+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a64340f0 0x5586a6436540 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16486f7501f96f54 server_cookie=e4c47530a8c56e50 in_seq=0 out_seq=0 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f57fff640 1 -- 172.21.15.105:0/3550146024 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586a6393880 con 0x5586a64340f0 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f57fff640 10 client.16308 handle_command_reply: tid=0 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 client.16308 shutdown 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 msgr2=0x5586a6420de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9f640ef670 tx=0x7f9f640bd8a0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 shutdown_connections 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a64340f0 0x5586a6436540 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a64240e0 0x5586a6421340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641b8d0 0x5586a6420de0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 --2- 172.21.15.105:0/3550146024 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641ab60 0x5586a6423a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 >> 172.21.15.105:0/3550146024 conn(0x5586a63e1c90 msgr2=0x5586a64175a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f57fff640 0 client.16308 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 shutdown_connections 2024-04-22T06:39:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 1 -- 172.21.15.105:0/3550146024 wait complete. 2024-04-22T06:39:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.428+0000 7f9f815fd740 20 client.16308 trim_cache size 0 max 16384 2024-04-22T06:39:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f815fd740 1 Processor -- start 2024-04-22T06:39:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f815fd740 1 -- start start 2024-04-22T06:39:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 0x5586a642fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 0x5586a6423510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.457+0000 7f9f7d312640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38466/0 (socket says 172.21.15.105:38466) 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f77fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 0x5586a6423510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 -- 172.21.15.105:0/4230234265 learned_addr learned my addr 172.21.15.105:0/4230234265 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781abfd0 con 0x5586a64240e0 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781adb10 con 0x5586a641b8d0 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781ac6a0 con 0x5586a641ab60 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 0x5586a642fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 -- 172.21.15.105:0/4230234265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 msgr2=0x5586a642fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 0x5586a642fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 -- 172.21.15.105:0/4230234265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 msgr2=0x5586a6423510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 0x5586a6423510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 -- 172.21.15.105:0/4230234265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586a6410f30 con 0x5586a64240e0 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 0x5586a642fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f77fff640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 0x5586a6423510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.458+0000 7f9f7d312640 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9f700129d0 tx=0x7f9f70029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f6371dd9635f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.459+0000 7f9f627fc640 1 -- 172.21.15.105:0/4230234265 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70004d20 con 0x5586a64240e0 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.459+0000 7f9f627fc640 1 -- 172.21.15.105:0/4230234265 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f70030040 con 0x5586a64240e0 2024-04-22T06:39:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.459+0000 7f9f627fc640 1 -- 172.21.15.105:0/4230234265 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7001d810 con 0x5586a64240e0 2024-04-22T06:39:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 msgr2=0x5586a6423a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9f700129d0 tx=0x7f9f70029000 comp rx=0 tx=0).stop 2024-04-22T06:39:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 shutdown_connections 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a64240e0 0x5586a6423a50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a641b8d0 0x5586a6423510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 --2- 172.21.15.105:0/4230234265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641ab60 0x5586a642fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 >> 172.21.15.105:0/4230234265 conn(0x5586a63e1c90 msgr2=0x5586a6415380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 shutdown_connections 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- 172.21.15.105:0/4230234265 wait complete. 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.460+0000 7f9f815fd740 1 -- start start 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 Processor -- start 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641b8d0 0x5586a641e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 0x5586a641d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781ad440 con 0x5586a641ab60 2024-04-22T06:39:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781acd70 con 0x5586a64240e0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781a9fb0 con 0x5586a641b8d0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f815fd740 10 client.0 _ll_register_callbacks cb 0x7f9f64009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f7cb11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f7cb11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38470/0 (socket says 172.21.15.105:38470) 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f7cb11640 1 -- 172.21.15.105:0/1023399047 learned_addr learned my addr 172.21.15.105:0/1023399047 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 0x5586a641d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.461+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641b8d0 0x5586a641e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 -- 172.21.15.105:0/1023399047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641b8d0 msgr2=0x5586a641e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641b8d0 0x5586a641e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 -- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 msgr2=0x5586a641d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 0x5586a641d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 -- 172.21.15.105:0/1023399047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586a641d5b0 con 0x5586a641ab60 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7d312640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 0x5586a641d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f7cb11640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x5586a6436df0 tx=0x7f9f6c01d410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6173edc07e58790 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f57fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c004220 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f9f6c0043c0 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f70030040 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9f6c005020 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f815fd740 10 client.16314 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5586a6499cf0 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.462+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f6c0047a0 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f9f6c003680 con 0x5586a641ab60 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 10 client.16314 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 20 client.16314 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a649a4a0 0x5586a649c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 4 client.16314 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9f70030040 con 0x5586a649a4a0 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a649a4a0 0x5586a649c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f57fff640 10 client.16314 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:04.463+0000 7f9f77fff640 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a649a4a0 0x5586a649c8c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7d6c5708b4e968 server_cookie=5fbd407804b3b523 in_seq=0 out_seq=0 2024-04-22T06:39:04.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:04 smithi105 ceph-mon[27916]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-22T06:39:04.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:04 smithi112 ceph-mon[31236]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-22T06:39:04.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:04 smithi177 ceph-mon[31346]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f57fff640 1 -- 172.21.15.105:0/1023399047 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9f70030040 con 0x5586a649a4a0 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f57fff640 10 client.16314 handle_command_reply: tid=0 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 client.16314 shutdown 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 msgr2=0x5586a642c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x5586a6436df0 tx=0x7f9f6c01d410 comp rx=0 tx=0).stop 2024-04-22T06:39:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 shutdown_connections 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x5586a649a4a0 0x5586a649c8c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5586a64240e0 0x5586a641d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x5586a641b8d0 0x5586a641e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 --2- 172.21.15.105:0/1023399047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5586a641ab60 0x5586a642c740 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 >> 172.21.15.105:0/1023399047 conn(0x5586a63e1c90 msgr2=0x5586a641f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f57fff640 0 client.16314 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 shutdown_connections 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.294+0000 7f9f815fd740 1 -- 172.21.15.105:0/1023399047 wait complete. 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f815fd740 20 client.16314 trim_cache size 0 max 16384 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:39:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:39:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:39:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:39:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 msgr2=0x7f9f680a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 0x7f9f680a6430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f7001cfd0 tx=0x7f9f7000e3a0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 msgr2=0x7f9f7819d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9f6402c240 tx=0x7f9f6402c7e0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 shutdown_connections 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9f680a3f70 0x7f9f680a6430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f78106170 0x7f9f7819d980 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 --2- 172.21.15.105:0/1340858180 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9f781010f0 0x7f9f7819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.295+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 >> 172.21.15.105:0/1340858180 conn(0x7f9f78077fd0 msgr2=0x7f9f78103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.296+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 shutdown_connections 2024-04-22T06:39:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:05.296+0000 7f9f7ed75640 1 -- 172.21.15.105:0/1340858180 wait complete. 2024-04-22T06:39:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:06 smithi105 ceph-mon[27916]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:06.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:06 smithi112 ceph-mon[31236]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:06.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:06 smithi177 ceph-mon[31346]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:08.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:08 smithi105 ceph-mon[27916]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:08.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:08 smithi112 ceph-mon[31236]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:08.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:08 smithi177 ceph-mon[31346]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:39:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:10 smithi105 ceph-mon[27916]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:10 smithi105 ceph-mon[27916]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:39:10.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:10 smithi112 ceph-mon[31236]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:10.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:10 smithi112 ceph-mon[31236]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:39:10.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:10 smithi177 ceph-mon[31346]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:10 smithi177 ceph-mon[31346]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-22T06:39:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:12 smithi105 ceph-mon[27916]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-22T06:39:12.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:12 smithi112 ceph-mon[31236]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-22T06:39:12.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:12 smithi177 ceph-mon[31346]: pgmap v54: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-22T06:39:14.160 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- 172.21.15.105:0/31854298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 msgr2=0x7f558c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 --2- 172.21.15.105:0/31854298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c101620 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5588009930 tx=0x7f558802bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- 172.21.15.105:0/31854298 shutdown_connections 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 --2- 172.21.15.105:0/31854298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c101b60 0x7f558c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 --2- 172.21.15.105:0/31854298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c101620 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.265 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 --2- 172.21.15.105:0/31854298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c105e30 0x7f558c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- 172.21.15.105:0/31854298 >> 172.21.15.105:0/31854298 conn(0x7f558c077e30 msgr2=0x7f558c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- 172.21.15.105:0/31854298 shutdown_connections 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- 172.21.15.105:0/31854298 wait complete. 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 Processor -- start 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 -- start start 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.265+0000 7f55933b1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 0x7f558c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 0x7f558c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f558c1110e0 con 0x7f558c1011a0 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f558c110f60 con 0x7f558c105e30 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f558c111260 con 0x7f558c101b60 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:54104/0 (socket says 172.21.15.105:54104) 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 -- 172.21.15.105:0/650985420 learned_addr learned my addr 172.21.15.105:0/650985420 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559094c640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 0x7f558c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559194e640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 0x7f558c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 -- 172.21.15.105:0/650985420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 msgr2=0x7f558c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 0x7f558c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 -- 172.21.15.105:0/650985420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 msgr2=0x7f558c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 0x7f558c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 -- 172.21.15.105:0/650985420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c19d740 con 0x7f558c1011a0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559094c640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 0x7f558c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559194e640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 0x7f558c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f559114d640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f557c00d770 tx=0x7f557c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c017070 con 0x7f558c1011a0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f557c0054d0 con 0x7f558c1011a0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c19da30 con 0x7f558c1011a0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c004c30 con 0x7f558c1011a0 2024-04-22T06:39:14.267 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.266+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c1a52c0 con 0x7f558c1011a0 2024-04-22T06:39:14.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.267+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c069230 con 0x7f558c1011a0 2024-04-22T06:39:14.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.269+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f557c004e10 con 0x7f558c1011a0 2024-04-22T06:39:14.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.270+0000 7f55827fc640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 0x7f55780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.270+0000 7f559094c640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 0x7f55780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.270+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f557c0d0470 con 0x7f558c1011a0 2024-04-22T06:39:14.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.270+0000 7f559094c640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 0x7f55780a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5588036040 tx=0x7f5588000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.273 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.274+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f557c09ca20 con 0x7f558c1011a0 2024-04-22T06:39:14.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.499+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f558c10a360 con 0x7f558c1011a0 2024-04-22T06:39:14.497 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.499+0000 7f55827fc640 1 -- 172.21.15.105:0/650985420 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11943 (secure 0 0 0) 0x7f557c09c320 con 0x7f558c1011a0 2024-04-22T06:39:14.498 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:39:14.498 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:38:41.031516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24431,"mds_2":24398,"mds_3":24397,"mds_4":14622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":31,"state":"up:active","state_seq":51,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":33,"state":"up:active","state_seq":55,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24397,14622,24398,24500,24431]},"id":1}]} 2024-04-22T06:39:14.498 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 37 2024-04-22T06:39:14.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 msgr2=0x7f55780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 0x7f55780a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5588036040 tx=0x7f5588000bc0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 msgr2=0x7f558c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f557c00d770 tx=0x7f557c00dc30 comp rx=0 tx=0).stop 2024-04-22T06:39:14.500 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 shutdown_connections 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f55780a3d20 0x7f55780a61e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f558c105e30 0x7f558c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f558c101b60 0x7f558c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 --2- 172.21.15.105:0/650985420 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f558c1011a0 0x7f558c198750 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 >> 172.21.15.105:0/650985420 conn(0x7f558c077e30 msgr2=0x7f558c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 shutdown_connections 2024-04-22T06:39:14.501 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.501+0000 7f55933b1640 1 -- 172.21.15.105:0/650985420 wait complete. 2024-04-22T06:39:14.511 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id 824d2692 --release --await' 2024-04-22T06:39:14.511 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db --set-id 824d2692 --release --await 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 -- 172.21.15.105:0/1573575583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 msgr2=0x7f532410eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 --2- 172.21.15.105:0/1573575583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532410eeb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5320009900 tx=0x7f532002bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 -- 172.21.15.105:0/1573575583 shutdown_connections 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 --2- 172.21.15.105:0/1573575583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532410eeb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 --2- 172.21.15.105:0/1573575583 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102bc0 0x7f5324106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 --2- 172.21.15.105:0/1573575583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5324102210 0x7f53241025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 -- 172.21.15.105:0/1573575583 >> 172.21.15.105:0/1573575583 conn(0x7f53240fdab0 msgr2=0x7f53240ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 -- 172.21.15.105:0/1573575583 shutdown_connections 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.619+0000 7f532b681640 1 -- 172.21.15.105:0/1573575583 wait complete. 2024-04-22T06:39:14.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 Processor -- start 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 -- start start 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102210 0x7f532419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53241986c0 0x7f5324198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5324112320 con 0x7f53241072e0 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53241121a0 con 0x7f5324102210 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532b681640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53241124a0 con 0x7f53241986c0 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f5328c1c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f5328c1c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55086/0 (socket says 172.21.15.105:55086) 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f5328c1c640 1 -- 172.21.15.105:0/4036714367 learned_addr learned my addr 172.21.15.105:0/4036714367 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f5329c1e640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53241986c0 0x7f5324198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.620+0000 7f532941d640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102210 0x7f532419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 -- 172.21.15.105:0/4036714367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53241986c0 msgr2=0x7f5324198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53241986c0 0x7f5324198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 -- 172.21.15.105:0/4036714367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102210 msgr2=0x7f532419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102210 0x7f532419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 -- 172.21.15.105:0/4036714367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53241992c0 con 0x7f53241072e0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f5328c1c640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f531400b450 tx=0x7f531400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f531a7fc640 1 -- 172.21.15.105:0/4036714367 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5314015020 con 0x7f53241072e0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f531a7fc640 1 -- 172.21.15.105:0/4036714367 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5314011d40 con 0x7f53241072e0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241a53a0 con 0x7f53241072e0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f531a7fc640 1 -- 172.21.15.105:0/4036714367 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400f340 con 0x7f53241072e0 2024-04-22T06:39:14.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.621+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241a57a0 con 0x7f53241072e0 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.623+0000 7f531a7fc640 1 -- 172.21.15.105:0/4036714367 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f5314018020 con 0x7f53241072e0 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 Processor -- start 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 -- start start 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f531a7fc640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 0x7f53100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc447216f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc4472cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bc44686500 con 0x55bc44721c30 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bc4462c6a0 con 0x55bc44721310 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532e27a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bc446824c0 con 0x55bc4472a810 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f531a7fc640 1 -- 172.21.15.105:0/4036714367 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f53140d02b0 con 0x7f53241072e0 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f5329c1e640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 0x7f53100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f5328c1c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f5328c1c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55094/0 (socket says 172.21.15.105:55094) 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f5328c1c640 1 -- 172.21.15.105:0/3965759265 learned_addr learned my addr 172.21.15.105:0/3965759265 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f5329c1e640 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc4472cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.624+0000 7f532941d640 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc447216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 -- 172.21.15.105:0/3965759265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 msgr2=0x55bc4472cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc4472cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 -- 172.21.15.105:0/3965759265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 msgr2=0x55bc447216f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc447216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 -- 172.21.15.105:0/3965759265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4464b5e0 con 0x55bc44721c30 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5328c1c640 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f531409bf70 tx=0x7f531409bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f74b37ccf97483d server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f5329c1e640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 0x7f53100a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5320005ab0 tx=0x7f53200040c0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f530f7fe640 1 -- 172.21.15.105:0/3965759265 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140a0620 con 0x55bc44721c30 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f530f7fe640 1 -- 172.21.15.105:0/3965759265 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53140a4ee0 con 0x55bc44721c30 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.625+0000 7f530f7fe640 1 -- 172.21.15.105:0/3965759265 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140546a0 con 0x55bc44721c30 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- 172.21.15.105:0/3965759265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 msgr2=0x55bc4472a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f531409bf70 tx=0x7f531409bcd0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- 172.21.15.105:0/3965759265 shutdown_connections 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc4472cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721c30 0x55bc4472a2d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- 172.21.15.105:0/3965759265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc447216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- 172.21.15.105:0/3965759265 >> 172.21.15.105:0/3965759265 conn(0x55bc4471ba40 msgr2=0x55bc4471be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- 172.21.15.105:0/3965759265 shutdown_connections 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- 172.21.15.105:0/3965759265 wait complete. 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- start start 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 Processor -- start 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721c30 0x55bc44727bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc44728100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bc44731a10 con 0x55bc44721310 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55bc4463fa60 con 0x55bc44721c30 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55bc44731b90 con 0x55bc4472a810 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532e27a740 10 client.0 _ll_register_callbacks cb 0x55bc44398110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.626+0000 7f532941d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55108/0 (socket says 172.21.15.105:55108) 2024-04-22T06:39:14.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 -- 172.21.15.105:0/2337703473 learned_addr learned my addr 172.21.15.105:0/2337703473 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 -- 172.21.15.105:0/2337703473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 msgr2=0x55bc44728100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f5329c1e640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc44728100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc44728100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f5328c1c640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721c30 0x55bc44727bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 -- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721c30 msgr2=0x55bc44727bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721c30 0x55bc44727bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 -- 172.21.15.105:0/2337703473 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc44398920 con 0x55bc44721310 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f5329c1e640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc44728100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532941d640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f531c00ac50 tx=0x7f531c007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617007f2fe75516f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f530cff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c01b060 con 0x55bc44721310 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f531c016640 con 0x55bc44721310 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53140a4ee0 con 0x55bc44721310 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f531c022050 con 0x55bc44721310 2024-04-22T06:39:14.632 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532e27a740 10 client.16332 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bc44464d30 con 0x55bc44721310 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.627+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53140a4ee0 con 0x55bc44721310 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f531c014340 con 0x55bc44721310 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 10 client.16332 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 10 client.16332 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 20 client.16332 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc4473ee50 0x55bc44741280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 4 client.16332 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc44659a10 con 0x55bc4473ee50 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f5328c1c640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc4473ee50 0x55bc44741280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f530cff9640 10 client.16332 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.628+0000 7f5328c1c640 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc4473ee50 0x55bc44741280 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f820715dd367afd3 server_cookie=db0f00dfffcc51f0 in_seq=0 out_seq=0 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f530cff9640 1 -- 172.21.15.105:0/2337703473 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc44659a10 con 0x55bc4473ee50 2024-04-22T06:39:14.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f530cff9640 10 client.16332 handle_command_reply: tid=0 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 client.16332 shutdown 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 msgr2=0x55bc44727680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f531c00ac50 tx=0x7f531c007680 comp rx=0 tx=0).stop 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 shutdown_connections 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc4473ee50 0x55bc44741280 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc4472a810 0x55bc44728100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721c30 0x55bc44727bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f530cff9640 0 client.16332 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 --2- 172.21.15.105:0/2337703473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44727680 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 >> 172.21.15.105:0/2337703473 conn(0x55bc4471ba40 msgr2=0x55bc4471d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 shutdown_connections 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.629+0000 7f532e27a740 1 -- 172.21.15.105:0/2337703473 wait complete. 2024-04-22T06:39:14.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.630+0000 7f532e27a740 20 client.16332 trim_cache size 0 max 16384 2024-04-22T06:39:14.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.646+0000 7f532e27a740 1 Processor -- start 2024-04-22T06:39:14.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.646+0000 7f532e27a740 1 -- start start 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44728110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc4472a810 0x55bc44726f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53241abb60 con 0x55bc44721310 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ad6a0 con 0x55bc4472a810 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532e27a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ac230 con 0x55bc44721c30 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55114/0 (socket says 172.21.15.105:55114) 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 -- 172.21.15.105:0/2865040939 learned_addr learned my addr 172.21.15.105:0/2865040939 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f5328c1c640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44728110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 -- 172.21.15.105:0/2865040939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 msgr2=0x55bc44728110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.650 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44728110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 -- 172.21.15.105:0/2865040939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc4472a810 msgr2=0x55bc44726f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f5329c1e640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc4472a810 0x55bc44726f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc4472a810 0x55bc44726f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 -- 172.21.15.105:0/2865040939 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc44721a60 con 0x55bc44721310 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f5328c1c640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44728110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f532941d640 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f531c00a510 tx=0x7f531c007fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25640053fb13bdef server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.647+0000 7f530effd640 1 -- 172.21.15.105:0/2865040939 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c02a070 con 0x55bc44721310 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f530effd640 1 -- 172.21.15.105:0/2865040939 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f531c014340 con 0x55bc44721310 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f530effd640 1 -- 172.21.15.105:0/2865040939 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c00ae60 con 0x55bc44721310 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 -- 172.21.15.105:0/2865040939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 msgr2=0x55bc44729db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f531c00a510 tx=0x7f531c007fc0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 -- 172.21.15.105:0/2865040939 shutdown_connections 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc4472a810 0x55bc44726f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44728110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 --2- 172.21.15.105:0/2865040939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc44721310 0x55bc44729db0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 -- 172.21.15.105:0/2865040939 >> 172.21.15.105:0/2865040939 conn(0x55bc4471ba40 msgr2=0x55bc4471d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 -- 172.21.15.105:0/2865040939 shutdown_connections 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.648+0000 7f532e27a740 1 -- 172.21.15.105:0/2865040939 wait complete. 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 -- start start 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 Processor -- start 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc44736fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44733db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f53241acfd0 con 0x55bc4472a810 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f53241ac900 con 0x55bc44721310 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f53241a9ba0 con 0x55bc44721c30 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532e27a740 10 client.0 _ll_register_callbacks cb 0x55bc4472a760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55116/0 (socket says 172.21.15.105:55116) 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 -- 172.21.15.105:0/556203252 learned_addr learned my addr 172.21.15.105:0/556203252 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:14.652 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f532941d640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc44736fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5328c1c640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44733db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 -- 172.21.15.105:0/556203252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 msgr2=0x55bc44733db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44733db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 -- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 msgr2=0x55bc44736fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc44736fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5329c1e640 1 -- 172.21.15.105:0/556203252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc44734830 con 0x55bc4472a810 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.649+0000 7f5328c1c640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44733db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f5329c1e640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f532002c780 tx=0x7f53200058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958f3105817c610b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320054020 con 0x55bc4472a810 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f5320004c10 con 0x55bc4472a810 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f531c014340 con 0x55bc4472a810 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5320004e10 con 0x55bc4472a810 2024-04-22T06:39:14.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 10 client.16338 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bc447a61a0 con 0x55bc4472a810 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f531c014340 con 0x55bc4472a810 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f5320043520 con 0x55bc4472a810 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 10 client.16338 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 10 client.16338 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 20 client.16338 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc447a6970 0x55bc447a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 4 client.16338 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "824d2692", "release": true, "await": true}] 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.650+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "set_id": "824d2692", "release": true, "await": true}) -- 0x55bc44693f20 con 0x55bc447a6970 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.651+0000 7f532941d640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc447a6970 0x55bc447a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.651+0000 7f532941d640 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc447a6970 0x55bc447a8d90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10a66962780830b4 server_cookie=2d8a360b639c33d6 in_seq=0 out_seq=0 2024-04-22T06:39:14.654 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.651+0000 7f530cff9640 10 client.16338 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:14.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f530cff9640 1 -- 172.21.15.105:0/556203252 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55bc44693f20 con 0x55bc447a6970 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f530cff9640 10 client.16338 handle_command_reply: tid=0 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 client.16338 shutdown 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 msgr2=0x55bc447342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f532002c780 tx=0x7f53200058d0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 shutdown_connections 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55bc447a6970 0x55bc447a8d90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bc4472a810 0x55bc447342f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f530cff9640 0 client.16338 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55bc44721c30 0x55bc44733db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 --2- 172.21.15.105:0/556203252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55bc44721310 0x55bc44736fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 >> 172.21.15.105:0/556203252 conn(0x55bc4471ba40 msgr2=0x55bc44725f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 shutdown_connections 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 1 -- 172.21.15.105:0/556203252 wait complete. 2024-04-22T06:39:14.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.653+0000 7f532e27a740 20 client.16338 trim_cache size 0 max 16384 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 37, 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 8, 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "824d2692": { 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 8, 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:39:14.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 77.5, 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:14.658 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 msgr2=0x7f53100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 0x7f53100a61e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5320005ab0 tx=0x7f53200040c0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 msgr2=0x7f532419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f531400b450 tx=0x7f531400b910 comp rx=0 tx=0).stop 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 shutdown_connections 2024-04-22T06:39:14.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f53100a3d20 0x7f53100a61e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f53241986c0 0x7f5324198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53241072e0 0x7f532419eb40 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 --2- 172.21.15.105:0/4036714367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5324102210 0x7f532419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 >> 172.21.15.105:0/4036714367 conn(0x7f53240fdab0 msgr2=0x7f53240fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 shutdown_connections 2024-04-22T06:39:14.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:14.654+0000 7f532b681640 1 -- 172.21.15.105:0/4036714367 wait complete. 2024-04-22T06:39:14.663 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 824d2692, seconds elapsed: 18.8 2024-04-22T06:39:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-mon[27916]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/650985420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fe4f0bef700 time 2024-04-22T06:39:14.652662+0000 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe50422a3dd] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: clone() 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.653+0000 7fe4f0bef700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fe4f0bef700 time 2024-04-22T06:39:14.652662+0000 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe50422a3dd] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.920 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: clone() 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: *** Caught signal (Aborted) ** 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: in thread 7fe4f0bef700 thread_name:quiesce.agt 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe502fb2d20] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: gsignal() 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: abort() 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe50422a437] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.921 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 13: clone() 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.655+0000 7fe4f0bef700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.922 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: in thread 7fe4f0bef700 thread_name:quiesce.agt 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe502fb2d20] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: gsignal() 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: abort() 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe50422a437] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.923 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 13: clone() 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: -1> 2024-04-22T06:39:14.653+0000 7fe4f0bef700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fe4f0bef700 time 2024-04-22T06:39:14.652662+0000 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe50422a3dd] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.924 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: clone() 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 0> 2024-04-22T06:39:14.655+0000 7fe4f0bef700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: in thread 7fe4f0bef700 thread_name:quiesce.agt 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe502fb2d20] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: gsignal() 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: abort() 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe50422a437] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.925 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 13: clone() 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: -4> 2024-04-22T06:39:14.653+0000 7fe4f0bef700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7fe4f0bef700 time 2024-04-22T06:39:14.652662+0000 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe50422a3dd] 2024-04-22T06:39:14.926 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: clone() 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: -3> 2024-04-22T06:39:14.655+0000 7fe4f0bef700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: in thread 7fe4f0bef700 thread_name:quiesce.agt 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe502fb2d20] 2024-04-22T06:39:14.927 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2: gsignal() 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 3: abort() 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe50422a437] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7fe50422a5a3] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562463fa5e9c] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562463fa5f9f] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562463ed40d3] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562463ed403c] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562464153d3f] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5624641bc560] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fe502fa81ca] 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 13: clone() 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[69837]: 2024-04-22T06:39:14.928 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:14 smithi105 podman[87108]: 2024-04-22 06:39:14.739228319 +0000 UTC m=+0.020018705 container died 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:39:14.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-mon[31236]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:14.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/650985420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f0540108700 time 2024-04-22T06:39:14.652773+0000 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05537433dd] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.959 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: clone() 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: *** Caught signal (Aborted) ** 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: in thread 7f0540108700 thread_name:quiesce.agt 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.652+0000 7f0540108700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f0540108700 time 2024-04-22T06:39:14.652773+0000 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.960 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f28dcb4e700 time 2024-04-22T06:39:14.652808+0000 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28f01893dd] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: clone() 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.652+0000 7f28dcb4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f28dcb4e700 time 2024-04-22T06:39:14.652808+0000 2024-04-22T06:39:14.961 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28f01893dd] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05537433dd] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.962 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: clone() 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05524cbd20] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: gsignal() 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: abort() 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0553743437] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.963 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 13: clone() 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.654+0000 7f0540108700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: in thread 7f0540108700 thread_name:quiesce.agt 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: clone() 2024-04-22T06:39:14.964 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: *** Caught signal (Aborted) ** 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: in thread 7f28dcb4e700 thread_name:quiesce.agt 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f28eef11d20] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: gsignal() 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: abort() 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28f0189437] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.965 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 13: clone() 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.654+0000 7f28dcb4e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: in thread 7f28dcb4e700 thread_name:quiesce.agt 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05524cbd20] 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: gsignal() 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: abort() 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0553743437] 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.966 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 13: clone() 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: -1> 2024-04-22T06:39:14.652+0000 7f0540108700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f0540108700 time 2024-04-22T06:39:14.652773+0000 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05537433dd] 2024-04-22T06:39:14.967 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f28eef11d20] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: gsignal() 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: abort() 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28f0189437] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.968 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 13: clone() 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: -1> 2024-04-22T06:39:14.652+0000 7f28dcb4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f28dcb4e700 time 2024-04-22T06:39:14.652808+0000 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28f01893dd] 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.969 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: clone() 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 0> 2024-04-22T06:39:14.654+0000 7f0540108700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: in thread 7f0540108700 thread_name:quiesce.agt 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.970 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05524cbd20] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: gsignal() 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: abort() 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0553743437] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 13: clone() 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.971 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: clone() 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 0> 2024-04-22T06:39:14.654+0000 7f28dcb4e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.972 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: in thread 7f28dcb4e700 thread_name:quiesce.agt 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f28eef11d20] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: gsignal() 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: abort() 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28f0189437] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 13: clone() 2024-04-22T06:39:14.973 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: -1> 2024-04-22T06:39:14.652+0000 7f28dcb4e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f28dcb4e700 time 2024-04-22T06:39:14.652808+0000 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28f01893dd] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.974 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: clone() 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 0> 2024-04-22T06:39:14.654+0000 7f28dcb4e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: in thread 7f28dcb4e700 thread_name:quiesce.agt 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f28eef11d20] 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2: gsignal() 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 3: abort() 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28f0189437] 2024-04-22T06:39:14.975 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28f01895a3] 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: -1> 2024-04-22T06:39:14.652+0000 7f0540108700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f0540108700 time 2024-04-22T06:39:14.652773+0000 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f05537433dd] 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.977 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: clone() 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 0> 2024-04-22T06:39:14.654+0000 7f0540108700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: in thread 7f0540108700 thread_name:quiesce.agt 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.978 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f05524cbd20] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2: gsignal() 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 3: abort() 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0553743437] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f05537435a3] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x556f308e0e9c] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x556f308e0f9f] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x556f3080f0d3] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x556f3080f03c] 2024-04-22T06:39:14.979 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x556f30a8ed3f] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x556f30af7560] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f28eef071ca] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 13: clone() 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[60419]: 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:14 smithi112 podman[64225]: 2024-04-22 06:39:14.732149644 +0000 UTC m=+0.023046716 container died bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x562ecef7ee9c] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x562ecef7ef9f] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x562eceead0d3] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x562eceead03c] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x562ecf12cd3f] 2024-04-22T06:39:14.980 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x562ecf195560] 2024-04-22T06:39:14.981 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f05524c11ca] 2024-04-22T06:39:14.981 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 13: clone() 2024-04-22T06:39:14.981 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.981 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[60098]: 2024-04-22T06:39:14.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-mon[31346]: pgmap v55: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 29 op/s 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/650985420' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8581d37700 time 2024-04-22T06:39:14.652993+0000 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f85953723dd] 2024-04-22T06:39:14.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: clone() 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: *** Caught signal (Aborted) ** 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: in thread 7f8581d37700 thread_name:quiesce.agt 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.653+0000 7f8581d37700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8581d37700 time 2024-04-22T06:39:14.652993+0000 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.991 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f85953723dd] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.992 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: clone() 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f85940fad20] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: gsignal() 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: abort() 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8595372437] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.993 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 13: clone() 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.654+0000 7f8581d37700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: in thread 7f8581d37700 thread_name:quiesce.agt 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f85940fad20] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: gsignal() 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: abort() 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8595372437] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.994 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 13: clone() 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: -1> 2024-04-22T06:39:14.653+0000 7f8581d37700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8581d37700 time 2024-04-22T06:39:14.652993+0000 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f85953723dd] 2024-04-22T06:39:14.995 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.996 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: clone() 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 0> 2024-04-22T06:39:14.654+0000 7f8581d37700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: in thread 7f8581d37700 thread_name:quiesce.agt 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f85940fad20] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: gsignal() 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: abort() 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8595372437] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.997 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 13: clone() 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: -1> 2024-04-22T06:39:14.653+0000 7f8581d37700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f8581d37700 time 2024-04-22T06:39:14.652993+0000 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f85953723dd] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:14.998 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: clone() 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 0> 2024-04-22T06:39:14.654+0000 7f8581d37700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: in thread 7f8581d37700 thread_name:quiesce.agt 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f85940fad20] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2: gsignal() 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 3: abort() 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8595372437] 2024-04-22T06:39:14.999 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f85953725a3] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x55e36bbd7e9c] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x55e36bbd7f9f] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x55e36bb060d3] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x55e36bb0603c] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x55e36bd85d3f] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x55e36bdee560] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f85940f01ca] 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 13: clone() 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[60992]: 2024-04-22T06:39:15.000 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:14 smithi177 podman[65371]: 2024-04-22 06:39:14.715820619 +0000 UTC m=+0.021751355 container died 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:39:15.113 INFO:tasks.workunit.client.0.smithi105.stderr:Note: switching to '86f7587a5a09af35f0895e1a2d08527638fae697'. 2024-04-22T06:39:15.113 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:state without impacting any branches by switching back to a branch. 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: git switch -c 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:Or undo this operation with: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: git switch - 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.114 INFO:tasks.workunit.client.0.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-22T06:39:15.115 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:15.115 INFO:tasks.workunit.client.0.smithi105.stderr:HEAD is now at 86f7587a5a0 qa/fsx: use the master branch to build the xfstest-dev 2024-04-22T06:39:15.119 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-22T06:39:15.196 INFO:tasks.workunit.client.0.smithi105.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-22T06:39:15.198 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:39:15.198 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-22T06:39:15.821 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:15 smithi112 podman[64225]: 2024-04-22 06:39:15.546675295 +0000 UTC m=+0.837572400 container remove bd5a881cbc55e9de5ccdf0aaf23f4ec7209d118b29a3517a5f5385069d60ac7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:39:15.822 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:15 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:39:15.822 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:15 smithi112 podman[64224]: 2024-04-22 06:39:15.564241221 +0000 UTC m=+0.855115068 container died 34e012d0142ea90aaac55059aa17ca425df3dc60b032810cddecada48c04ab23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:39:15.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:15 smithi177 podman[65371]: 2024-04-22 06:39:15.642243045 +0000 UTC m=+0.948173765 container remove 75448994ff9ae7618b572c78da12e9ea974c8dc1b476c706178e26beb0ae35c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:39:15.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:15 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:39:15.990 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:15 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-22T06:39:16.168 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:15 smithi105 podman[87108]: 2024-04-22 06:39:15.775381032 +0000 UTC m=+1.056171407 container remove 21f47b3cd334f97ac7bcdb015a9a04d5e44743a8e74a3316069e7fb326fa2867 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-22T06:39:16.168 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:15 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:39:16.168 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:16 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-22T06:39:16.235 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-22T06:39:16.289 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-22T06:39:16.329 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-22T06:39:16.330 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:39:16.330 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-22T06:39:16.366 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:16 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-22T06:39:16.366 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:16 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.001s CPU time. 2024-04-22T06:39:16.366 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:16 smithi112 podman[64224]: 2024-04-22 06:39:16.115594985 +0000 UTC m=+1.406468877 container remove 34e012d0142ea90aaac55059aa17ca425df3dc60b032810cddecada48c04ab23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T06:39:16.366 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:16 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:39:16.369 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-22T06:39:16.373 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:39:16.373 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-22T06:39:16.389 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-04-22T06:39:16.390 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-04-22T06:39:16.390 DEBUG:teuthology.orchestra.run.smithi105:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=86f7587a5a09af35f0895e1a2d08527638fae697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-04-22T06:39:16.450 INFO:tasks.workunit.client.0.smithi105.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-04-22T06:39:16.479 INFO:tasks.workunit.client.0.smithi105.stderr:--2024-04-22 06:39:16-- http://download.ceph.com/qa/fsync-tester.c 2024-04-22T06:39:16.480 INFO:tasks.workunit.client.0.smithi105.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-22T06:39:16.519 INFO:tasks.workunit.client.0.smithi105.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-22T06:39:16.557 INFO:tasks.workunit.client.0.smithi105.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T06:39:16.558 INFO:tasks.workunit.client.0.smithi105.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-04-22T06:39:16.558 INFO:tasks.workunit.client.0.smithi105.stderr:Saving to: ‘fsync-tester.c’ 2024-04-22T06:39:16.562 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:16.563 INFO:tasks.workunit.client.0.smithi105.stderr: 0K .. 100% 564M=0s 2024-04-22T06:39:16.563 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:16.563 INFO:tasks.workunit.client.0.smithi105.stderr:2024-04-22 06:39:16 (564 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-04-22T06:39:16.563 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-04-22T06:39:16.563 INFO:tasks.workunit.client.0.smithi105.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-04-22T06:39:16.630 INFO:tasks.workunit.client.0.smithi105.stderr:+ ./fsync-tester 2024-04-22T06:39:16.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:16 smithi105 ceph-mon[27916]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:16.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:16 smithi112 ceph-mon[31236]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:16.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:16 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-22T06:39:16.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:16 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.018s CPU time. 2024-04-22T06:39:16.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:16 smithi177 ceph-mon[31346]: pgmap v56: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 -- 172.21.15.105:0/3561091278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c1015c0 msgr2=0x7f3d0c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/3561091278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c101a20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009900 tx=0x7f3cfc02bc80 comp rx=0 tx=0).stop 2024-04-22T06:39:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 -- 172.21.15.105:0/3561091278 shutdown_connections 2024-04-22T06:39:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/3561091278 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c101f60 0x7f3d0c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/3561091278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c101a20 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/3561091278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 0x7f3d0c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 -- 172.21.15.105:0/3561091278 >> 172.21.15.105:0/3561091278 conn(0x7f3d0c0773e0 msgr2=0x7f3d0c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 -- 172.21.15.105:0/3561091278 shutdown_connections 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.391+0000 7f3d13ad2640 1 -- 172.21.15.105:0/3561091278 wait complete. 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.392+0000 7f3d13ad2640 1 Processor -- start 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.392+0000 7f3d13ad2640 1 -- start start 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.392+0000 7f3d13ad2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 0x7f3d0c1a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c111190 con 0x7f3d0c101f60 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c111010 con 0x7f3d0c105e20 2024-04-22T06:39:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c111310 con 0x7f3d0c1015c0 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55140/0 (socket says 172.21.15.105:55140) 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 -- 172.21.15.105:0/2555237946 learned_addr learned my addr 172.21.15.105:0/2555237946 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1186e640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1206f640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 0x7f3d0c1a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 -- 172.21.15.105:0/2555237946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c1015c0 msgr2=0x7f3d0c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 -- 172.21.15.105:0/2555237946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 msgr2=0x7f3d0c1a46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 0x7f3d0c1a46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 -- 172.21.15.105:0/2555237946 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0c1a4bf0 con 0x7f3d0c101f60 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d1106d640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc005940 tx=0x7f3cfc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d02ffd640 1 -- 172.21.15.105:0/2555237946 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc043020 con 0x7f3d0c101f60 2024-04-22T06:39:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d02ffd640 1 -- 172.21.15.105:0/2555237946 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cfc0054c0 con 0x7f3d0c101f60 2024-04-22T06:39:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d0c1a4dc0 con 0x7f3d0c101f60 2024-04-22T06:39:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.393+0000 7f3d02ffd640 1 -- 172.21.15.105:0/2555237946 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc003d40 con 0x7f3d0c101f60 2024-04-22T06:39:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.394+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d0c1a5220 con 0x7f3d0c101f60 2024-04-22T06:39:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.396+0000 7f3d1679f740 1 Processor -- start 2024-04-22T06:39:18.401 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.396+0000 7f3d1679f740 1 -- start start 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c266fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c2724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d02ffd640 1 -- 172.21.15.105:0/2555237946 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f3cfc046020 con 0x7f3d0c101f60 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b3c1cb8d0 con 0x558b3c2678d0 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558b3c168e80 con 0x558b3c2700e0 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1679f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558b3c1c7890 con 0x558b3c266b60 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55152/0 (socket says 172.21.15.105:55152) 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 -- 172.21.15.105:0/2955036503 learned_addr learned my addr 172.21.15.105:0/2955036503 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1186e640 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c266fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 -- 172.21.15.105:0/2955036503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 msgr2=0x558b3c266fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.402 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c266fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 -- 172.21.15.105:0/2955036503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 msgr2=0x558b3c2724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1106d640 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c2724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c2724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 -- 172.21.15.105:0/2955036503 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b3c18a320 con 0x558b3c2678d0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1206f640 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3d04008dc0 tx=0x7f3d0400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c74d51882dc885 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d02ffd640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 0x7f3cf80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3d1186e640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 0x7f3cf80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.397+0000 7f3cf7fff640 1 -- 172.21.15.105:0/2955036503 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04014070 con 0x558b3c2678d0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d02ffd640 1 -- 172.21.15.105:0/2555237946 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3cfc0f28f0 con 0x7f3d0c101f60 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3cf7fff640 1 -- 172.21.15.105:0/2955036503 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d04003b70 con 0x558b3c2678d0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3cf7fff640 1 -- 172.21.15.105:0/2955036503 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04010790 con 0x558b3c2678d0 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 -- 172.21.15.105:0/2955036503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 msgr2=0x558b3c266620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1186e640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 0x7f3cf80a6280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08005d20 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3d04008dc0 tx=0x7f3d0400e720 comp rx=0 tx=0).stop 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 -- 172.21.15.105:0/2955036503 shutdown_connections 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c2724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c266fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 --2- 172.21.15.105:0/2955036503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c266620 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.398+0000 7f3d1679f740 1 -- 172.21.15.105:0/2955036503 >> 172.21.15.105:0/2955036503 conn(0x558b3c22dc90 msgr2=0x558b3c261850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- 172.21.15.105:0/2955036503 shutdown_connections 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- 172.21.15.105:0/2955036503 wait complete. 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- start start 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 Processor -- start 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c266b60 0x558b3c26faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 0x558b3c26ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x558b3c272e10 con 0x558b3c2700e0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558b3c153eb0 con 0x558b3c266b60 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x558b3c272f90 con 0x558b3c2678d0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1679f740 10 client.0 _ll_register_callbacks cb 0x558b3c180b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1206f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1206f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55168/0 (socket says 172.21.15.105:55168) 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.399+0000 7f3d1206f640 1 -- 172.21.15.105:0/1938225751 learned_addr learned my addr 172.21.15.105:0/1938225751 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1106d640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 0x558b3c26ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1186e640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c266b60 0x558b3c26faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 -- 172.21.15.105:0/1938225751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 msgr2=0x558b3c26ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 0x558b3c26ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 -- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c266b60 msgr2=0x558b3c26faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c266b60 0x558b3c26faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.405 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 -- 172.21.15.105:0/1938225751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3bede210 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1106d640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 0x558b3c26ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1206f640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d04004540 tx=0x7f3d04004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d42239c4d59cd97 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04027060 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3d04010510 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d04003b70 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3d0402e050 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1679f740 10 client.16350 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.400+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558b3bfaa620 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d04003b70 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f3d040227b0 con 0x558b3c2700e0 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 10 client.16350 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:18.406 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 20 client.16350 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2800b0 0x558b3c2824a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 4 client.16350 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b3c1df880 con 0x558b3c2800b0 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1186e640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2800b0 0x558b3c2824a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3cf57fa640 10 client.16350 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.401+0000 7f3d1186e640 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2800b0 0x558b3c2824a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6474f634e3cdb7f6 server_cookie=ebd0b832f89b1bc4 in_seq=0 out_seq=0 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3cf57fa640 1 -- 172.21.15.105:0/1938225751 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b3c1df880 con 0x558b3c2800b0 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3cf57fa640 10 client.16350 handle_command_reply: tid=0 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 client.16350 shutdown 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 msgr2=0x558b3c26d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d04004540 tx=0x7f3d04004570 comp rx=0 tx=0).stop 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 shutdown_connections 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2800b0 0x558b3c2824a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.407 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2700e0 0x558b3c26d440 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2678d0 0x558b3c26ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3cf57fa640 0 client.16350 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.402+0000 7f3d1679f740 1 --2- 172.21.15.105:0/1938225751 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c266b60 0x558b3c26faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.403+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 >> 172.21.15.105:0/1938225751 conn(0x558b3c22dc90 msgr2=0x558b3c2635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.403+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 shutdown_connections 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.403+0000 7f3d1679f740 1 -- 172.21.15.105:0/1938225751 wait complete. 2024-04-22T06:39:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.403+0000 7f3d1679f740 20 client.16350 trim_cache size 0 max 16384 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.417+0000 7f3d1679f740 1 Processor -- start 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.417+0000 7f3d1679f740 1 -- start start 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c27bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c26faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1adb00 con 0x558b3c2678d0 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1af640 con 0x558b3c2700e0 2024-04-22T06:39:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1679f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1ae1d0 con 0x558b3c266b60 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55174/0 (socket says 172.21.15.105:55174) 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 -- 172.21.15.105:0/3253463547 learned_addr learned my addr 172.21.15.105:0/3253463547 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1186e640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c27bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1206f640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c26faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 -- 172.21.15.105:0/3253463547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 msgr2=0x558b3c27bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c27bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 -- 172.21.15.105:0/3253463547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 msgr2=0x558b3c26faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c26faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 -- 172.21.15.105:0/3253463547 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b3c265e30 con 0x558b3c2678d0 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1206f640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c26faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3d1106d640 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0f0550 tx=0x7f3cfc0044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c7aee484393da2 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3cf77fe640 1 -- 172.21.15.105:0/3253463547 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0be680 con 0x558b3c2678d0 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.418+0000 7f3cf77fe640 1 -- 172.21.15.105:0/3253463547 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cfc0befb0 con 0x558b3c2678d0 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3cf77fe640 1 -- 172.21.15.105:0/3253463547 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc0be210 con 0x558b3c2678d0 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 -- 172.21.15.105:0/3253463547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 msgr2=0x558b3c26f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc0f0550 tx=0x7f3cfc0044e0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 -- 172.21.15.105:0/3253463547 shutdown_connections 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2700e0 0x558b3c26faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c2678d0 0x558b3c26f560 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 --2- 172.21.15.105:0/3253463547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c266b60 0x558b3c27bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 -- 172.21.15.105:0/3253463547 >> 172.21.15.105:0/3253463547 conn(0x558b3c22dc90 msgr2=0x558b3c2634c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 -- 172.21.15.105:0/3253463547 shutdown_connections 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.419+0000 7f3d1679f740 1 -- 172.21.15.105:0/3253463547 wait complete. 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 -- start start 2024-04-22T06:39:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 Processor -- start 2024-04-22T06:39:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2678d0 0x558b3c278770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2700e0 0x558b3c2739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1aef70 con 0x558b3c266b60 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1ae8a0 con 0x558b3c2678d0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c1abb00 con 0x558b3c2700e0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1679f740 10 client.0 _ll_register_callbacks cb 0x7f3d04008c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1186e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1186e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55176/0 (socket says 172.21.15.105:55176) 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1186e640 1 -- 172.21.15.105:0/364041405 learned_addr learned my addr 172.21.15.105:0/364041405 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1106d640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2678d0 0x558b3c278770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1206f640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2700e0 0x558b3c2739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.420+0000 7f3d1186e640 1 -- 172.21.15.105:0/364041405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2700e0 msgr2=0x558b3c2739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1186e640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2700e0 0x558b3c2739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1186e640 1 -- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2678d0 msgr2=0x558b3c278770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1186e640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2678d0 0x558b3c278770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1186e640 1 -- 172.21.15.105:0/364041405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3c2741a0 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1186e640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x558b3c282a10 tx=0x7f3d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3842f9f40944d98a server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011910 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cfc0befb0 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f3d08011ab0 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3d08011cb0 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 10 client.16356 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558b3c2e6000 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cfc0befb0 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f3d0802d070 con 0x558b3c266b60 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 10 client.16356 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 10 client.16356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 20 client.16356 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.421+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2e67b0 0x558b3c2e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.422+0000 7f3d1679f740 4 client.16356 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.422+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b3c16b3d0 con 0x558b3c2e67b0 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.422+0000 7f3d1106d640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2e67b0 0x558b3c2e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.422+0000 7f3cf57fa640 10 client.16356 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.422+0000 7f3d1106d640 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2e67b0 0x558b3c2e8bd0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a7aaddb6e75a24 server_cookie=f2448ae09ae4d109 in_seq=0 out_seq=0 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3cf57fa640 1 -- 172.21.15.105:0/364041405 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b3c16b3d0 con 0x558b3c2e67b0 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3cf57fa640 10 client.16356 handle_command_reply: tid=0 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 client.16356 shutdown 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 msgr2=0x558b3c278230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x558b3c282a10 tx=0x7f3d080046c0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 shutdown_connections 2024-04-22T06:39:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x558b3c2e67b0 0x558b3c2e8bd0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x558b3c2700e0 0x558b3c2739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3cf57fa640 0 client.16356 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b3c2678d0 0x558b3c278770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 --2- 172.21.15.105:0/364041405 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558b3c266b60 0x558b3c278230 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 >> 172.21.15.105:0/364041405 conn(0x558b3c22dc90 msgr2=0x558b3c262c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 shutdown_connections 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.432+0000 7f3d1679f740 1 -- 172.21.15.105:0/364041405 wait complete. 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d1679f740 20 client.16356 trim_cache size 0 max 16384 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:39:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 msgr2=0x7f3cf80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 0x7f3cf80a6280 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3d08005d90 tx=0x7f3d08005d20 comp rx=0 tx=0).stop 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 msgr2=0x7f3d0c199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc005940 tx=0x7f3cfc0042f0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 shutdown_connections 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3cf80a3dc0 0x7f3cf80a6280 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d0c105e20 0x7f3d0c1a46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d0c101f60 0x7f3d0c199b70 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 --2- 172.21.15.105:0/2555237946 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d0c1015c0 0x7f3d0c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 >> 172.21.15.105:0/2555237946 conn(0x7f3d0c0773e0 msgr2=0x7f3d0c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 shutdown_connections 2024-04-22T06:39:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:18.433+0000 7f3d13ad2640 1 -- 172.21.15.105:0/2555237946 wait complete. 2024-04-22T06:39:18.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:18 smithi105 ceph-mon[27916]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:18.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:18 smithi112 ceph-mon[31236]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:18.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:18 smithi177 ceph-mon[31346]: pgmap v57: 97 pgs: 97 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-22T06:39:19.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-22T06:39:19.548 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-22T06:39:19.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- 172.21.15.105:0/57211778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406af60 msgr2=0x7f4654106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 --2- 172.21.15.105:0/57211778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406af60 0x7f4654106b90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4644009900 tx=0x7f464402bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- 172.21.15.105:0/57211778 shutdown_connections 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 --2- 172.21.15.105:0/57211778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 0x7f465410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 --2- 172.21.15.105:0/57211778 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406af60 0x7f4654106b90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 --2- 172.21.15.105:0/57211778 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f465406a5b0 0x7f465406a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- 172.21.15.105:0/57211778 >> 172.21.15.105:0/57211778 conn(0x7f4654077c40 msgr2=0x7f4654078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- 172.21.15.105:0/57211778 shutdown_connections 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- 172.21.15.105:0/57211778 wait complete. 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 Processor -- start 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.659+0000 7f465aec8640 1 -- start start 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 0x7f465419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46541985b0 0x7f4654198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46541121f0 con 0x7f465406a5b0 2024-04-22T06:39:19.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4654112070 con 0x7f4654107270 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4654112370 con 0x7f46541985b0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55202/0 (socket says 172.21.15.105:55202) 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 -- 172.21.15.105:0/940232919 learned_addr learned my addr 172.21.15.105:0/940232919 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4659465640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46541985b0 0x7f4654198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f464bfff640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 0x7f465419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 -- 172.21.15.105:0/940232919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46541985b0 msgr2=0x7f4654198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46541985b0 0x7f4654198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 -- 172.21.15.105:0/940232919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 msgr2=0x7f465419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 0x7f465419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 -- 172.21.15.105:0/940232919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46541990d0 con 0x7f465406a5b0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4658c64640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f465000b4a0 tx=0x7f465000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f4649ffb640 1 -- 172.21.15.105:0/940232919 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650014070 con 0x7f465406a5b0 2024-04-22T06:39:19.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.660+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46541993c0 con 0x7f465406a5b0 2024-04-22T06:39:19.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.661+0000 7f4649ffb640 1 -- 172.21.15.105:0/940232919 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4650010ac0 con 0x7f465406a5b0 2024-04-22T06:39:19.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.661+0000 7f4649ffb640 1 -- 172.21.15.105:0/940232919 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650017c30 con 0x7f465406a5b0 2024-04-22T06:39:19.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.661+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46541a56e0 con 0x7f465406a5b0 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.663+0000 7f4649ffb640 1 -- 172.21.15.105:0/940232919 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f4650017070 con 0x7f465406a5b0 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 Processor -- start 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 -- start start 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eab2b0 0x55fc18eab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4649ffb640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 0x7f463c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ebb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fc18e101e0 con 0x55fc18eabbd0 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc18dad390 con 0x55fc18eb8e90 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f465db98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc18e0c1a0 con 0x55fc18eab2b0 2024-04-22T06:39:19.669 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f464bfff640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 0x7f463c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4649ffb640 1 -- 172.21.15.105:0/940232919 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f46500cfe20 con 0x7f465406a5b0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55214/0 (socket says 172.21.15.105:55214) 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 -- 172.21.15.105:0/2622218083 learned_addr learned my addr 172.21.15.105:0/2622218083 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4659465640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ebb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4659465640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eab2b0 0x55fc18eab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 -- 172.21.15.105:0/2622218083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eab2b0 msgr2=0x55fc18eab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eab2b0 0x55fc18eab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 -- 172.21.15.105:0/2622218083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 msgr2=0x55fc18ebb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ebb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4658c64640 1 -- 172.21.15.105:0/2622218083 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc18bec7d0 con 0x55fc18eabbd0 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.664+0000 7f4659465640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ebb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:19.670 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.665+0000 7f464bfff640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 0x7f463c0a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f4644005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.665+0000 7f4658c64640 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f465000b470 tx=0x7f465009b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67c71b54e2c3be0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.665+0000 7f463affd640 1 -- 172.21.15.105:0/2622218083 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001e070 con 0x55fc18eabbd0 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.665+0000 7f463affd640 1 -- 172.21.15.105:0/2622218083 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46500a0190 con 0x55fc18eabbd0 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.665+0000 7f463affd640 1 -- 172.21.15.105:0/2622218083 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46500cd700 con 0x55fc18eabbd0 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- 172.21.15.105:0/2622218083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 msgr2=0x55fc18eb8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f465000b470 tx=0x7f465009b7f0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- 172.21.15.105:0/2622218083 shutdown_connections 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ebb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb8950 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- 172.21.15.105:0/2622218083 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eab2b0 0x55fc18eab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- 172.21.15.105:0/2622218083 >> 172.21.15.105:0/2622218083 conn(0x55fc18ea5cb0 msgr2=0x55fc18ea6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- 172.21.15.105:0/2622218083 shutdown_connections 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- 172.21.15.105:0/2622218083 wait complete. 2024-04-22T06:39:19.671 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- start start 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 Processor -- start 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 0x55fc18eb64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18eb69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55fc18ebbc50 con 0x55fc18eab2b0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fc18dc92f0 con 0x55fc18eabbd0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.666+0000 7f465db98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55fc18ebbdd0 con 0x55fc18eb8e90 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f465db98740 10 client.0 _ll_register_callbacks cb 0x55fc18dd6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55222/0 (socket says 172.21.15.105:55222) 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 -- 172.21.15.105:0/390026260 learned_addr learned my addr 172.21.15.105:0/390026260 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4659465640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18eb69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f464bfff640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 0x55fc18eb64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 -- 172.21.15.105:0/390026260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 msgr2=0x55fc18eb69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.672 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18eb69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 -- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 msgr2=0x55fc18eb64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 0x55fc18eb64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 -- 172.21.15.105:0/390026260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc18b223f0 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f464bfff640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 0x55fc18eb64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4658c64640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4650009420 tx=0x7f465009c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce83b5e11829210 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4633fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001e050 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f46500cd700 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46500a0190 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f46500770d0 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f465db98740 10 client.16368 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.667+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fc18ae20f0 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46500a0190 con 0x55fc18eab2b0 2024-04-22T06:39:19.673 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f46500a45c0 con 0x55fc18eab2b0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 10 client.16368 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 20 client.16368 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18ec8de0 0x55fc18ecb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 4 client.16368 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc18e1dc00 con 0x55fc18ec8de0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.668+0000 7f464bfff640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18ec8de0 0x55fc18ecb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f4633fff640 10 client.16368 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f464bfff640 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18ec8de0 0x55fc18ecb1d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5eb75c4935766b0 server_cookie=14c4819c8f20e3b1 in_seq=0 out_seq=0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f4633fff640 1 -- 172.21.15.105:0/390026260 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc18e1dc00 con 0x55fc18ec8de0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f4633fff640 10 client.16368 handle_command_reply: tid=0 2024-04-22T06:39:19.674 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 client.16368 shutdown 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 msgr2=0x55fc18eb5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f4650009420 tx=0x7f465009c010 comp rx=0 tx=0).stop 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 shutdown_connections 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18ec8de0 0x55fc18ecb1d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18eb69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f4633fff640 0 client.16368 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.669+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eabbd0 0x55fc18eb64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.670+0000 7f465db98740 1 --2- 172.21.15.105:0/390026260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18eb5f70 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.670+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 >> 172.21.15.105:0/390026260 conn(0x55fc18ea5cb0 msgr2=0x55fc18ea7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.670+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 shutdown_connections 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.670+0000 7f465db98740 1 -- 172.21.15.105:0/390026260 wait complete. 2024-04-22T06:39:19.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.670+0000 7f465db98740 20 client.16368 trim_cache size 0 max 16384 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.687+0000 7f465db98740 1 Processor -- start 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.687+0000 7f465db98740 1 -- start start 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eab2b0 0x55fc18eb86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18ece1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46541abc80 con 0x55fc18eabbd0 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46541ad7c0 con 0x55fc18eab2b0 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f465db98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46541ac350 con 0x55fc18eb8e90 2024-04-22T06:39:19.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55238/0 (socket says 172.21.15.105:55238) 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 -- 172.21.15.105:0/635162486 learned_addr learned my addr 172.21.15.105:0/635162486 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f4659465640 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18ece1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f4658c64640 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eab2b0 0x55fc18eb86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 -- 172.21.15.105:0/635162486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 msgr2=0x55fc18ece1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18ece1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 -- 172.21.15.105:0/635162486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eab2b0 msgr2=0x55fc18eb86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eab2b0 0x55fc18eb86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 -- 172.21.15.105:0/635162486 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc18ea57e0 con 0x55fc18eabbd0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f464bfff640 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4644009230 tx=0x7f464403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d6b6ec11af6f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.688+0000 7f463a7fc640 1 -- 172.21.15.105:0/635162486 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644005100 con 0x55fc18eabbd0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f463a7fc640 1 -- 172.21.15.105:0/635162486 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4644057070 con 0x55fc18eabbd0 2024-04-22T06:39:19.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f463a7fc640 1 -- 172.21.15.105:0/635162486 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644004920 con 0x55fc18eabbd0 2024-04-22T06:39:19.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 -- 172.21.15.105:0/635162486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 msgr2=0x55fc18eb5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f4644009230 tx=0x7f464403e000 comp rx=0 tx=0).stop 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 -- 172.21.15.105:0/635162486 shutdown_connections 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eb8e90 0x55fc18ece1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eabbd0 0x55fc18eb5850 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 --2- 172.21.15.105:0/635162486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eab2b0 0x55fc18eb86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 -- 172.21.15.105:0/635162486 >> 172.21.15.105:0/635162486 conn(0x55fc18ea5cb0 msgr2=0x55fc18ec48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 -- 172.21.15.105:0/635162486 shutdown_connections 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.689+0000 7f465db98740 1 -- 172.21.15.105:0/635162486 wait complete. 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 -- start start 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 Processor -- start 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eabbd0 0x55fc18ecb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ecbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46541ad0f0 con 0x55fc18eab2b0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46541aca20 con 0x55fc18eb8e90 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f46541a9c80 con 0x55fc18eabbd0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f465db98740 10 client.0 _ll_register_callbacks cb 0x7f464c00ac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f4658c64640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55252/0 (socket says 172.21.15.105:55252) 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f4658c64640 1 -- 172.21.15.105:0/744309996 learned_addr learned my addr 172.21.15.105:0/744309996 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f464bfff640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eabbd0 0x55fc18ecb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.690+0000 7f4659465640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ecbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 -- 172.21.15.105:0/744309996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eabbd0 msgr2=0x55fc18ecb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eabbd0 0x55fc18ecb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 -- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 msgr2=0x55fc18ecbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:19.694 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ecbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 -- 172.21.15.105:0/744309996 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc18ecc170 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4658c64640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f465009b830 tx=0x7f46500047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a625f0be712bca2f server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465001e070 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7f465009b330 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4644057070 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f46500a47c0 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 10 client.16374 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fc18ecc170 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4644057070 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7f46500cd6d0 con 0x55fc18eab2b0 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 10 client.16374 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 20 client.16374 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:19.695 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18f2f100 0x55fc18f31520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 4 client.16374 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.691+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc18e17a70 con 0x55fc18f2f100 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f464bfff640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18f2f100 0x55fc18f31520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f4633fff640 10 client.16374 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f464bfff640 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18f2f100 0x55fc18f31520 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=295a4b1fa88b8352 server_cookie=928e82904a041125 in_seq=0 out_seq=0 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f4633fff640 1 -- 172.21.15.105:0/744309996 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fc18e17a70 con 0x55fc18f2f100 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f4633fff640 10 client.16374 handle_command_reply: tid=0 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 client.16374 shutdown 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 msgr2=0x55fc18ece310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f465009b830 tx=0x7f46500047d0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 shutdown_connections 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55fc18f2f100 0x55fc18f31520 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fc18eb8e90 0x55fc18ecbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55fc18eabbd0 0x55fc18ecb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 --2- 172.21.15.105:0/744309996 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fc18eab2b0 0x55fc18ece310 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 >> 172.21.15.105:0/744309996 conn(0x55fc18ea5cb0 msgr2=0x55fc18eb6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.692+0000 7f4633fff640 0 client.16374 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 shutdown_connections 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465db98740 1 -- 172.21.15.105:0/744309996 wait complete. 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465db98740 20 client.16374 trim_cache size 0 max 16384 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stdout: "status": "no active scrubs running", 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stdout: "scrubs": {} 2024-04-22T06:39:19.697 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:39:19.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 msgr2=0x7f463c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 0x7f463c0a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4644000c00 tx=0x7f4644005fd0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 msgr2=0x7f465419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f465000b4a0 tx=0x7f465000b960 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 shutdown_connections 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f463c0a3d20 0x7f463c0a61e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f46541985b0 0x7f4654198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4654107270 0x7f465419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 --2- 172.21.15.105:0/940232919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f465406a5b0 0x7f465419e400 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 >> 172.21.15.105:0/940232919 conn(0x7f4654077c40 msgr2=0x7f46540ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.693+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 shutdown_connections 2024-04-22T06:39:19.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.694+0000 7f465aec8640 1 -- 172.21.15.105:0/940232919 wait complete. 2024-04-22T06:39:19.704 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-22T06:39:19.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 -- 172.21.15.105:0/2414490007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 msgr2=0x7efee0106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 --2- 172.21.15.105:0/2414490007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0106210 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efed0009930 tx=0x7efed002bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 -- 172.21.15.105:0/2414490007 shutdown_connections 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 --2- 172.21.15.105:0/2414490007 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee0101b60 0x7efee010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 --2- 172.21.15.105:0/2414490007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee01011a0 0x7efee0101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 --2- 172.21.15.105:0/2414490007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0106210 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 -- 172.21.15.105:0/2414490007 >> 172.21.15.105:0/2414490007 conn(0x7efee0077e30 msgr2=0x7efee0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:19.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 -- 172.21.15.105:0/2414490007 shutdown_connections 2024-04-22T06:39:19.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.811+0000 7efee51b6640 1 -- 172.21.15.105:0/2414490007 wait complete. 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 Processor -- start 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 -- start start 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee01011a0 0x7efee01973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee0101b60 0x7efee0197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efee0111140 con 0x7efee0105e30 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efee0110fc0 con 0x7efee0101b60 2024-04-22T06:39:19.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.812+0000 7efee51b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7efee01112c0 con 0x7efee01011a0 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55266/0 (socket says 172.21.15.105:55266) 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 -- 172.21.15.105:0/408832378 learned_addr learned my addr 172.21.15.105:0/408832378 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efed7fff640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee0101b60 0x7efee0197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efeded9d640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee01011a0 0x7efee01973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 -- 172.21.15.105:0/408832378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee01011a0 msgr2=0x7efee01973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee01011a0 0x7efee01973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 -- 172.21.15.105:0/408832378 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee0101b60 msgr2=0x7efee0197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:19.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee0101b60 0x7efee0197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:19.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 -- 172.21.15.105:0/408832378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee01a6c40 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedf59e640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efed800b970 tx=0x7efed800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8019070 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efed8014440 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee01a6f30 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed80075d0 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.813+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee01a7450 con 0x7efee0105e30 2024-04-22T06:39:19.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.815+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee0102570 con 0x7efee0105e30 2024-04-22T06:39:19.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.816+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7efed801c070 con 0x7efee0105e30 2024-04-22T06:39:19.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.817+0000 7efedcd99640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 0x7efecc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:19.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.817+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7efed80cfd10 con 0x7efee0105e30 2024-04-22T06:39:19.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.817+0000 7efeded9d640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 0x7efecc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:19.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.817+0000 7efeded9d640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 0x7efecc0a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efed0006010 tx=0x7efed002c8c0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:19.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:19.822+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efed809c3d0 con 0x7efee0105e30 2024-04-22T06:39:20.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.046+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efee002d5c0 con 0x7efee0105e30 2024-04-22T06:39:20.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.047+0000 7efedcd99640 1 -- 172.21.15.105:0/408832378 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11943 (secure 0 0 0) 0x7efed809bcd0 con 0x7efee0105e30 2024-04-22T06:39:20.045 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:39:20.045 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:38:41.031516+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24431,"mds_2":24398,"mds_3":24397,"mds_4":14622},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14622":{"gid":14622,"name":"g","rank":4,"incarnation":31,"state":"up:active","state_seq":51,"addr":"172.21.15.105:6839/256609249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":256609249},{"type":"v1","addr":"172.21.15.105:6839","nonce":256609249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":3,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.177:6833/624977252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":624977252},{"type":"v1","addr":"172.21.15.177:6833","nonce":624977252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24398":{"gid":24398,"name":"k","rank":2,"incarnation":29,"state":"up:active","state_seq":57,"addr":"172.21.15.112:6833/583171810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":583171810},{"type":"v1","addr":"172.21.15.112:6833","nonce":583171810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":1,"incarnation":33,"state":"up:active","state_seq":55,"addr":"172.21.15.112:6835/3883368202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":3883368202},{"type":"v1","addr":"172.21.15.112:6835","nonce":3883368202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24397,14622,24398,24500,24431]},"id":1}]} 2024-04-22T06:39:20.045 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 37 2024-04-22T06:39:20.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 msgr2=0x7efecc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.048 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 0x7efecc0a61e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efed0006010 tx=0x7efed002c8c0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 msgr2=0x7efee0197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7efed800b970 tx=0x7efed800bd70 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 shutdown_connections 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7efecc0a3d20 0x7efecc0a61e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efee0105e30 0x7efee0197e50 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efee0101b60 0x7efee0197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 --2- 172.21.15.105:0/408832378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efee01011a0 0x7efee01973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 >> 172.21.15.105:0/408832378 conn(0x7efee0077e30 msgr2=0x7efee00fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 shutdown_connections 2024-04-22T06:39:20.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.050+0000 7efee51b6640 1 -- 172.21.15.105:0/408832378 wait complete. 2024-04-22T06:39:20.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 -- 172.21.15.105:0/3405251419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f81017d0 msgr2=0x7fd9f8101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 --2- 172.21.15.105:0/3405251419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f81017d0 0x7fd9f8101bb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009900 tx=0x7fd9e802bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 -- 172.21.15.105:0/3405251419 shutdown_connections 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 --2- 172.21.15.105:0/3405251419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9f8106380 0x7fd9f810df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 --2- 172.21.15.105:0/3405251419 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9f8102180 0x7fd9f8105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 --2- 172.21.15.105:0/3405251419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f81017d0 0x7fd9f8101bb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 -- 172.21.15.105:0/3405251419 >> 172.21.15.105:0/3405251419 conn(0x7fd9f8077410 msgr2=0x7fd9f80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 -- 172.21.15.105:0/3405251419 shutdown_connections 2024-04-22T06:39:20.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.173+0000 7fd9fd359640 1 -- 172.21.15.105:0/3405251419 wait complete. 2024-04-22T06:39:20.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 Processor -- start 2024-04-22T06:39:20.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 -- start start 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9f81017d0 0x7fd9f80fc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8106380 0x7fd9f80fcd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9f80fd2a0 0x7fd9f81a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f81112e0 con 0x7fd9f8106380 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8111160 con 0x7fd9f81017d0 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9fd359640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8111460 con 0x7fd9f80fd2a0 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9f67fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8106380 0x7fd9f80fcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9f67fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8106380 0x7fd9f80fcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55276/0 (socket says 172.21.15.105:55276) 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.174+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2881435847 learned_addr learned my addr 172.21.15.105:0/2881435847 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9f80fd2a0 0x7fd9f81a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9f81017d0 0x7fd9f80fc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2881435847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9f80fd2a0 msgr2=0x7fd9f81a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd9f80fd2a0 0x7fd9f81a6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2881435847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9f81017d0 msgr2=0x7fd9f80fc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9f81017d0 0x7fd9f80fc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2881435847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f81a72c0 con 0x7fd9f8106380 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9f8106380 0x7fd9f80fcd60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e9f0 tx=0x7fd9ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9dffff640 1 -- 172.21.15.105:0/2881435847 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00ccc0 con 0x7fd9f8106380 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9fd359640 1 -- 172.21.15.105:0/2881435847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f81a75b0 con 0x7fd9f8106380 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9dffff640 1 -- 172.21.15.105:0/2881435847 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ec0054d0 con 0x7fd9f8106380 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9fd359640 1 -- 172.21.15.105:0/2881435847 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f81a7b00 con 0x7fd9f8106380 2024-04-22T06:39:20.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.175+0000 7fd9dffff640 1 -- 172.21.15.105:0/2881435847 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec010430 con 0x7fd9f8106380 2024-04-22T06:39:20.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 Processor -- start 2024-04-22T06:39:20.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 -- start start 2024-04-22T06:39:20.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9dffff640 1 -- 172.21.15.105:0/2881435847 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fd9ec005020 con 0x7fd9f8106380 2024-04-22T06:39:20.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0f5e80 0x564f1b0fe430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0fea00 0x564f1b100e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564f1b05a1f0 con 0x564f1b0f54d0 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f1aff74b0 con 0x564f1b0f5e80 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fda0002e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564f1b0561b0 con 0x564f1b0fea00 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55282/0 (socket says 172.21.15.105:55282) 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 -- 172.21.15.105:0/2461180580 learned_addr learned my addr 172.21.15.105:0/2461180580 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0fea00 0x564f1b100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0f5e80 0x564f1b0fe430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9dffff640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd9e00a3d50 0x7fd9e00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 -- 172.21.15.105:0/2461180580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0fea00 msgr2=0x564f1b100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0fea00 0x564f1b100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 -- 172.21.15.105:0/2461180580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0f5e80 msgr2=0x564f1b0fe430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0f5e80 0x564f1b0fe430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.178+0000 7fd9f77fe640 1 -- 172.21.15.105:0/2461180580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f1afe91b0 con 0x564f1b0f54d0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9dffff640 1 -- 172.21.15.105:0/2881435847 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fd9ec0d0c20 con 0x7fd9f8106380 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009f90 tx=0x7fd9f0010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cba5146457ecd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd9e00a3d50 0x7fd9e00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9dd7fa640 1 -- 172.21.15.105:0/2461180580 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0022070 con 0x564f1b0f54d0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9dd7fa640 1 -- 172.21.15.105:0/2461180580 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9f001b3e0 con 0x564f1b0f54d0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.179+0000 7fd9dd7fa640 1 -- 172.21.15.105:0/2461180580 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f001a580 con 0x564f1b0f54d0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- 172.21.15.105:0/2461180580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 msgr2=0x564f1b0f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009f90 tx=0x7fd9f0010860 comp rx=0 tx=0).stop 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2881435847 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fd9e00a3d50 0x7fd9e00a6210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd9f001f000 tx=0x7fd9f0020000 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- 172.21.15.105:0/2461180580 shutdown_connections 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0fea00 0x564f1b100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0f5e80 0x564f1b0fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- 172.21.15.105:0/2461180580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f54d0 0x564f1b0f58b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- 172.21.15.105:0/2461180580 >> 172.21.15.105:0/2461180580 conn(0x564f1b0efba0 msgr2=0x564f1b0eff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- 172.21.15.105:0/2461180580 shutdown_connections 2024-04-22T06:39:20.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- 172.21.15.105:0/2461180580 wait complete. 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- start start 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 Processor -- start 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0f54d0 0x564f1b0fe330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f5e80 0x564f1b0fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0fea00 0x564f1b0fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564f1b101970 con 0x564f1b0f5e80 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564f1b013500 con 0x564f1b0fea00 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x564f1b101af0 con 0x564f1b0f54d0 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.180+0000 7fda0002e740 10 client.0 _ll_register_callbacks cb 0x564f1afe1b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f5e80 0x564f1b0fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f5e80 0x564f1b0fb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55284/0 (socket says 172.21.15.105:55284) 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2236332774 learned_addr learned my addr 172.21.15.105:0/2236332774 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0f54d0 0x564f1b0fe330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f77fe640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0fea00 0x564f1b0fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0f54d0 msgr2=0x564f1b0fe330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:20.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x564f1b0f54d0 0x564f1b0fe330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0fea00 msgr2=0x564f1b0fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564f1b0fea00 0x564f1b0fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 -- 172.21.15.105:0/2236332774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f1ad6c3f0 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9f67fc640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564f1b0f5e80 0x564f1b0fb6c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0ce2e0 tx=0x7fd9ec098760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea9542e4308f1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9daff5640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9daff5640 1 -- 172.21.15.105:0/2236332774 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec09c430 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fda0002e740 1 -- 172.21.15.105:0/2236332774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9f001b3e0 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9daff5640 1 -- 172.21.15.105:0/2236332774 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8755+0+0 (secure 0 0 0) 0x7fd9ec09add0 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9daff5640 1 -- 172.21.15.105:0/2236332774 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd9ec005310 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fda0002e740 10 client.16392 fetch_fsmap learned FSMap version 37 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fda0002e740 1 -- 172.21.15.105:0/2236332774 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564f1ae367d0 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.181+0000 7fd9daff5640 1 -- 172.21.15.105:0/2236332774 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9f001b3e0 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fd9daff5640 1 -- 172.21.15.105:0/2236332774 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 37) ==== 8354+0+0 (secure 0 0 0) 0x7fd9ec0b8050 con 0x564f1b0f5e80 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 10 client.16392 fetch_fsmap finished waiting for FSMap version 37 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 10 client.16392 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.g 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 20 client.16392 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:20.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:20.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 4 client.16392 mds_command: new command op to 14622 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:20.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fda0002e740 1 -- 172.21.15.105:0/2236332774 --> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f1aff9930 con 0x564f1b10ec30 2024-04-22T06:39:20.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.182+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-22T06:39:20.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.383+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.383+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-22T06:39:20.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.383+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.782 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.784+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.782 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.784+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-22T06:39:20.782 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:20.784+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:20 smithi105 ceph-mon[27916]: pgmap v58: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 179 KiB/s wr, 16 op/s 2024-04-22T06:39:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:20 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/408832378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:20 smithi112 ceph-mon[31236]: pgmap v58: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 179 KiB/s wr, 16 op/s 2024-04-22T06:39:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:20 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/408832378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:20 smithi177 ceph-mon[31346]: pgmap v58: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 179 KiB/s wr, 16 op/s 2024-04-22T06:39:20.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:20 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/408832378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:21.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:21.585+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:21.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:21.585+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-22T06:39:21.583 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:21.585+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:22 smithi105 ceph-mon[27916]: pgmap v59: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 177 KiB/s wr, 10 op/s 2024-04-22T06:39:22.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:22 smithi112 ceph-mon[31236]: pgmap v59: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 177 KiB/s wr, 10 op/s 2024-04-22T06:39:22.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:22 smithi177 ceph-mon[31346]: pgmap v59: 97 pgs: 97 active+clean; 2.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 177 KiB/s wr, 10 op/s 2024-04-22T06:39:23.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:23.186+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:23.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:23.186+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-22T06:39:23.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:23.186+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:24.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:24 smithi105 ceph-mon[27916]: pgmap v60: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 611 KiB/s wr, 18 op/s 2024-04-22T06:39:24.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:24 smithi112 ceph-mon[31236]: pgmap v60: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 611 KiB/s wr, 18 op/s 2024-04-22T06:39:24.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:24 smithi177 ceph-mon[31346]: pgmap v60: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 611 KiB/s wr, 18 op/s 2024-04-22T06:39:26.320 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:25 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:39:26.321 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:25 smithi177 systemd[1]: Stopped Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:26.321 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 systemd[1]: Starting Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:39:26.321 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 podman[65472]: 2024-04-22 06:39:26.225964441 +0000 UTC m=+0.112559836 container create 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:39:26.321 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 podman[65472]: 2024-04-22 06:39:26.131590417 +0000 UTC m=+0.018185835 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:39:26.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:26.390+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.105:6838/256609249 2024-04-22T06:39:26.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:26.390+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-22T06:39:26.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:26.390+0000 7fd9daff5640 1 client.16392 ms_handle_refused on v2:172.21.15.105:6838/256609249 2024-04-22T06:39:26.533 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:39:26.533 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 systemd[1]: Stopped Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:26.533 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 systemd[1]: Starting Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:39:26.533 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 podman[87504]: 2024-04-22 06:39:26.310581096 +0000 UTC m=+0.025276629 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:39:26.534 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 podman[87504]: 2024-04-22 06:39:26.419545497 +0000 UTC m=+0.134241012 container create 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T06:39:26.716 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:39:26.716 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: Stopped Ceph mds.k for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.018s CPU time. 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: Starting Ceph mds.k for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: Stopped Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.001s CPU time. 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 systemd[1]: Starting Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: pgmap v61: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 611 KiB/s wr, 14 op/s 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: daemon mds.l restarted 2024-04-22T06:39:26.717 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:boot 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:39:26.718 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:26 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: pgmap v61: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 611 KiB/s wr, 14 op/s 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: daemon mds.l restarted 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:boot 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 podman[65472]: 2024-04-22 06:39:26.436812148 +0000 UTC m=+0.323407542 container init 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:39:26.740 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 podman[65472]: 2024-04-22 06:39:26.439661084 +0000 UTC m=+0.326256479 container start 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-22T06:39:26.741 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-mds[65493]: -- 172.21.15.177:0/2338964951 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561b7f80da40 con 0x561b7f7e0c00 2024-04-22T06:39:26.741 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: starting mds.l at 2024-04-22T06:39:26.741 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 bash[65472]: 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f 2024-04-22T06:39:26.741 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:39:26 smithi177 systemd[1]: Started Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: pgmap v61: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 611 KiB/s wr, 14 op/s 2024-04-22T06:39:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: daemon mds.l restarted 2024-04-22T06:39:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:39:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: osdmap e97: 12 total, 12 up, 12 in 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:boot 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 podman[87504]: 2024-04-22 06:39:26.638823793 +0000 UTC m=+0.353519308 container init 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 podman[87504]: 2024-04-22 06:39:26.641752557 +0000 UTC m=+0.356448081 container start 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-mds[87524]: -- 172.21.15.105:0/157608677 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa150a1a40 con 0x55aa15074800 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: starting mds.g at 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 bash[87504]: 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 2024-04-22T06:39:26.919 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:39:26 smithi105 systemd[1]: Started Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:27.062 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 podman[64413]: 2024-04-22 06:39:26.622873261 +0000 UTC m=+0.020690877 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:39:27.062 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:26 smithi112 podman[64413]: 2024-04-22 06:39:26.760141562 +0000 UTC m=+0.157959201 container create 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:39:27.062 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 podman[64412]: 2024-04-22 06:39:26.624631608 +0000 UTC m=+0.022449299 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:39:27.062 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:26 smithi112 podman[64412]: 2024-04-22 06:39:26.960714223 +0000 UTC m=+0.358531899 container create 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T06:39:27.324 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 podman[64413]: 2024-04-22 06:39:27.17188724 +0000 UTC m=+0.569704849 container init 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:39:27.325 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 podman[64413]: 2024-04-22 06:39:27.174530406 +0000 UTC m=+0.572348004 container start 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-22T06:39:27.325 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mds[64449]: -- 172.21.15.112:0/918511977 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560643495a40 con 0x560643469000 2024-04-22T06:39:27.325 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: starting mds.e at 2024-04-22T06:39:27.325 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 bash[64413]: 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 2024-04-22T06:39:27.325 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:39:27 smithi112 systemd[1]: Started Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:27.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: daemon mds.g restarted 2024-04-22T06:39:27.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:39:27.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:boot 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,3=h=up:replay} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: daemon mds.e restarted 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:resolve 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:resolve 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:boot 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:active,3=h=up:resolve,4=a=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: daemon mds.k restarted 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:39:27.669 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:boot 2024-04-22T06:39:27.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:39:27.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:replay,3=h=up:resolve,4=a=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-22T06:39:27.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:39:27.670 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:27 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-04-22T06:39:27.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 podman[64412]: 2024-04-22 06:39:27.398350501 +0000 UTC m=+0.796168179 container init 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:39:27.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 podman[64412]: 2024-04-22 06:39:27.401227471 +0000 UTC m=+0.799045161 container start 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:39:27.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mds[64472]: -- 172.21.15.112:0/1196904709 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56092c261a40 con 0x56092c235000 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: starting mds.k at 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 bash[64412]: 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:39:27 smithi112 systemd[1]: Started Ceph mds.k for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: daemon mds.g restarted 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:boot 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,3=h=up:replay} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-22T06:39:27.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: daemon mds.e restarted 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:resolve 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:resolve 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:boot 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:active,3=h=up:resolve,4=a=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: daemon mds.k restarted 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:boot 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:replay,3=h=up:resolve,4=a=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-22T06:39:27.710 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:39:27.711 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:27 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: daemon mds.g restarted 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: osdmap e98: 12 total, 12 up, 12 in 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:boot 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=e=up:active,2=k=up:active,3=h=up:replay} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:39:27.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: daemon mds.e restarted 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: osdmap e99: 12 total, 12 up, 12 in 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:resolve 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:resolve 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:boot 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,2=k=up:active,3=h=up:resolve,4=a=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: daemon mds.k restarted 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: osdmap e100: 12 total, 12 up, 12 in 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:boot 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:replay,3=h=up:resolve,4=a=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-04-22T06:39:27.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:39:27.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:27 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 2 up:replay 2 up:resolve 1 up:active 2024-04-22T06:39:28.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:28 smithi112 ceph-mon[31236]: pgmap v64: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 651 KiB/s wr, 13 op/s 2024-04-22T06:39:28.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:28 smithi177 ceph-mon[31346]: pgmap v64: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 651 KiB/s wr, 13 op/s 2024-04-22T06:39:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:28 smithi105 ceph-mon[27916]: pgmap v64: 97 pgs: 97 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 651 KiB/s wr, 13 op/s 2024-04-22T06:39:29.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:29 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:resolve 2024-04-22T06:39:29.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:29 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:resolve 2024-04-22T06:39:29.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:29 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:39:29.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:29 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:resolve 2024-04-22T06:39:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:29 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:resolve 2024-04-22T06:39:29.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:29 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:39:30.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:29 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:resolve 2024-04-22T06:39:30.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:29 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:resolve 2024-04-22T06:39:30.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:29 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:39:30.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: pgmap v67: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:reconnect 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:reconnect 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:reconnect 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:reconnect 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00699977 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:rejoin 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:rejoin 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:rejoin 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:rejoin 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:39:30.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:39:30.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:39:30.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-22T06:39:30.960 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:30 smithi112 ceph-mon[31236]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: pgmap v67: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:reconnect 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:reconnect 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:reconnect 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:reconnect 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00699977 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:rejoin 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:rejoin 2024-04-22T06:39:30.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:rejoin 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:rejoin 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-22T06:39:30.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:30 smithi177 ceph-mon[31346]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: pgmap v67: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:reconnect 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:reconnect 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:reconnect 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:reconnect 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0.00699977 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:rejoin 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:rejoin 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:rejoin 2024-04-22T06:39:31.043 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:rejoin 2024-04-22T06:39:31.044 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:39:31.044 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-22T06:39:31.044 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-22T06:39:31.044 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-22T06:39:31.044 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:30 smithi105 ceph-mon[27916]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-22T06:39:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/2907078631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 msgr2=0x7f2dbc10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/2907078631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc10ede0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009cd0 tx=0x7f2db402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/2907078631 shutdown_connections 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/2907078631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc10ede0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/2907078631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 0x7f2dbc1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/2907078631 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 0x7f2dbc107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/2907078631 >> 172.21.15.105:0/2907078631 conn(0x7f2dbc0773e0 msgr2=0x7f2dbc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/2907078631 shutdown_connections 2024-04-22T06:39:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.530+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/2907078631 wait complete. 2024-04-22T06:39:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 Processor -- start 2024-04-22T06:39:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 -- start start 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 0x7f2dbc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 0x7f2dbc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc112290 con 0x7f2dbc100e10 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc112110 con 0x7f2dbc100450 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dc0cfd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc112410 con 0x7f2dbc106f30 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2db9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2db9d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38154/0 (socket says 172.21.15.105:38154) 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2db9d9b640 1 -- 172.21.15.105:0/1825680022 learned_addr learned my addr 172.21.15.105:0/1825680022 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 0x7f2dbc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.531+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 0x7f2dbc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 -- 172.21.15.105:0/1825680022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 msgr2=0x7f2dbc19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 0x7f2dbc19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 -- 172.21.15.105:0/1825680022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 msgr2=0x7f2dbc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 0x7f2dbc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 -- 172.21.15.105:0/1825680022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dbc19d6c0 con 0x7f2dbc100e10 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00e9f0 tx=0x7f2dac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2dab7fe640 1 -- 172.21.15.105:0/1825680022 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac00cd80 con 0x7f2dbc100e10 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2dab7fe640 1 -- 172.21.15.105:0/1825680022 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dac0108e0 con 0x7f2dbc100e10 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dbc19d9b0 con 0x7f2dbc100e10 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2dab7fe640 1 -- 172.21.15.105:0/1825680022 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac018980 con 0x7f2dbc100e10 2024-04-22T06:39:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.532+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dbc1a5240 con 0x7f2dbc100e10 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.534+0000 7f2dab7fe640 1 -- 172.21.15.105:0/1825680022 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f2dac010430 con 0x7f2dbc100e10 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.535+0000 7f2dab7fe640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 0x7f2da40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.535+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 0x7f2da40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.535+0000 7f2dab7fe640 1 -- 172.21.15.105:0/1825680022 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f2dac0d0db0 con 0x7f2dbc100e10 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.535+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 0x7f2da40a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2db0005fd0 tx=0x7f2db0005850 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 Processor -- start 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 -- start start 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c101b60 0x55e58c101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e58c0668d0 con 0x55e58c1028d0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e58c003e80 con 0x55e58c101b60 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e58c062890 con 0x55e58c10b0e0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38158/0 (socket says 172.21.15.105:38158) 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dba59c640 1 -- 172.21.15.105:0/3024132556 learned_addr learned my addr 172.21.15.105:0/3024132556 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c101b60 0x55e58c101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.536+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 -- 172.21.15.105:0/3024132556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 msgr2=0x55e58c10d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 -- 172.21.15.105:0/3024132556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c101b60 msgr2=0x55e58c101fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c101b60 0x55e58c101fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 -- 172.21.15.105:0/3024132556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e58c025320 con 0x55e58c1028d0 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2dba59c640 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2db000feb0 tx=0x7f2db0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d41306c10b719c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2da8ff9640 1 -- 172.21.15.105:0/3024132556 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0022070 con 0x55e58c1028d0 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2da8ff9640 1 -- 172.21.15.105:0/3024132556 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db0011e50 con 0x55e58c1028d0 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.537+0000 7f2da8ff9640 1 -- 172.21.15.105:0/3024132556 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0010950 con 0x55e58c1028d0 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- 172.21.15.105:0/3024132556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 msgr2=0x55e58c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2db000feb0 tx=0x7f2db0012000 comp rx=0 tx=0).stop 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- 172.21.15.105:0/3024132556 shutdown_connections 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c101b60 0x55e58c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/3024132556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c1028d0 0x55e58c101620 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- 172.21.15.105:0/3024132556 >> 172.21.15.105:0/3024132556 conn(0x55e58c0c8c90 msgr2=0x55e58c0fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- 172.21.15.105:0/3024132556 shutdown_connections 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- 172.21.15.105:0/3024132556 wait complete. 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 -- start start 2024-04-22T06:39:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.538+0000 7f2dc39ca740 1 Processor -- start 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c107d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c1082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e58c10de10 con 0x55e58c101b60 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55e58bfeeeb0 con 0x55e58c1028d0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55e58c10df90 con 0x55e58c10b0e0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 10 client.0 _ll_register_callbacks cb 0x55e58c01bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38172/0 (socket says 172.21.15.105:38172) 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 -- 172.21.15.105:0/578973400 learned_addr learned my addr 172.21.15.105:0/578973400 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c107d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 -- 172.21.15.105:0/578973400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 msgr2=0x55e58c1082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c1082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.545 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c1082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 -- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 msgr2=0x55e58c107d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c107d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 -- 172.21.15.105:0/578973400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e58bd79210 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c107d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dba59c640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2db0020040 tx=0x7f2db0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63471ea01777a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2da1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0036060 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f2db00046c0 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.539+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2db0011e50 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f2db003d050 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 10 client.16422 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e58be45620 con 0x55e58c101b60 2024-04-22T06:39:31.546 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db0011e50 con 0x55e58c101b60 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f2db0010ad0 con 0x55e58c101b60 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 10 client.16422 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 20 client.16422 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c11d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 4 client.16422 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e58c07a880 con 0x55e58c11b000 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.540+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c11d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c11d3f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=358dbbb90208a8b5 server_cookie=2711d5116b3036ef in_seq=0 out_seq=0 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2da1ffb640 10 client.16422 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2da1ffb640 1 -- 172.21.15.105:0/578973400 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e58c07a880 con 0x55e58c11b000 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2da1ffb640 10 client.16422 handle_command_reply: tid=0 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 client.16422 shutdown 2024-04-22T06:39:31.547 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 msgr2=0x55e58c10a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2db0020040 tx=0x7f2db0005270 comp rx=0 tx=0).stop 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 shutdown_connections 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c11d3f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.541+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c107d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2da1ffb640 0 client.16422 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/578973400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c10a9f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 >> 172.21.15.105:0/578973400 conn(0x55e58c0c8c90 msgr2=0x55e58c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 shutdown_connections 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 1 -- 172.21.15.105:0/578973400 wait complete. 2024-04-22T06:39:31.548 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.542+0000 7f2dc39ca740 20 client.16422 trim_cache size 0 max 16384 2024-04-22T06:39:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 Processor -- start 2024-04-22T06:39:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 -- start start 2024-04-22T06:39:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c0fb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1adfd0 con 0x55e58c101b60 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1afb10 con 0x55e58c1028d0 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1ae6a0 con 0x55e58c10b0e0 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38186/0 (socket says 172.21.15.105:38186) 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 -- 172.21.15.105:0/391997135 learned_addr learned my addr 172.21.15.105:0/391997135 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c0fb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 -- 172.21.15.105:0/391997135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 msgr2=0x55e58c10a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 -- 172.21.15.105:0/391997135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 msgr2=0x55e58c0fb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.558 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c0fb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.557+0000 7f2dba59c640 1 -- 172.21.15.105:0/391997135 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e58c100e30 con 0x55e58c101b60 2024-04-22T06:39:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2dba59c640 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2db000fbd0 tx=0x7f2db00052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579c406cc823c6f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2da3fff640 1 -- 172.21.15.105:0/391997135 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0036070 con 0x55e58c101b60 2024-04-22T06:39:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2da3fff640 1 -- 172.21.15.105:0/391997135 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db00055f0 con 0x55e58c101b60 2024-04-22T06:39:31.559 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2da3fff640 1 -- 172.21.15.105:0/391997135 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0015040 con 0x55e58c101b60 2024-04-22T06:39:31.562 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2dc39ca740 1 -- 172.21.15.105:0/391997135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 msgr2=0x55e58c106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2db000fbd0 tx=0x7f2db00052e0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2dc39ca740 1 -- 172.21.15.105:0/391997135 shutdown_connections 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.558+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c10b0e0 0x55e58c10a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c0fb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/391997135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c101b60 0x55e58c106ee0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- 172.21.15.105:0/391997135 >> 172.21.15.105:0/391997135 conn(0x55e58c0c8c90 msgr2=0x55e58c0fb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- 172.21.15.105:0/391997135 shutdown_connections 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- 172.21.15.105:0/391997135 wait complete. 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- start start 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 Processor -- start 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 0x55e58c1130c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c113600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1af440 con 0x55e58c10b0e0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1aed70 con 0x55e58c1028d0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc1abfd0 con 0x55e58c101b60 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dc39ca740 10 client.0 _ll_register_callbacks cb 0x7f2dac098770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dbad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dbad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:38188/0 (socket says 172.21.15.105:38188) 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dbad9d640 1 -- 172.21.15.105:0/1960073670 learned_addr learned my addr 172.21.15.105:0/1960073670 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2db9d9b640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.559+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 0x55e58c1130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 -- 172.21.15.105:0/1960073670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 msgr2=0x55e58c1130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 0x55e58c1130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 -- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 msgr2=0x55e58c113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 -- 172.21.15.105:0/1960073670 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e58c10ede0 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 0x55e58c1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dbad9d640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2db402c700 tx=0x7f2db402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25329fcda869035b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4039040 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f2db4032dc0 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2db00055f0 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f2db4053050 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 10 client.16428 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e58c180d60 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db00055f0 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f2db4004040 con 0x55e58c10b0e0 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 10 client.16428 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 10 client.16428 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 20 client.16428 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c183550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 4 client.16428 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.560+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e58c0063d0 con 0x55e58c11b000 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.561+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c183550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.561+0000 7f2dba59c640 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c183550 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f541b0138230bba server_cookie=b704fab7f3bf6a1c in_seq=0 out_seq=0 2024-04-22T06:39:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.561+0000 7f2da1ffb640 10 client.16428 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.571+0000 7f2da1ffb640 1 -- 172.21.15.105:0/1960073670 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e58c0063d0 con 0x55e58c11b000 2024-04-22T06:39:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.571+0000 7f2da1ffb640 10 client.16428 handle_command_reply: tid=0 2024-04-22T06:39:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.571+0000 7f2dc39ca740 1 client.16428 shutdown 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 msgr2=0x55e58c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2db402c700 tx=0x7f2db402cd30 comp rx=0 tx=0).stop 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 shutdown_connections 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55e58c11b000 0x55e58c183550 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e58c10b0e0 0x55e58c10e5b0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55e58c1028d0 0x55e58c113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2da1ffb640 0 client.16428 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 --2- 172.21.15.105:0/1960073670 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55e58c101b60 0x55e58c1130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 >> 172.21.15.105:0/1960073670 conn(0x55e58c0c8c90 msgr2=0x55e58c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 shutdown_connections 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 1 -- 172.21.15.105:0/1960073670 wait complete. 2024-04-22T06:39:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc39ca740 20 client.16428 trim_cache size 0 max 16384 2024-04-22T06:39:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:39:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:39:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:39:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 msgr2=0x7f2da40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 0x7f2da40a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2db0005fd0 tx=0x7f2db0005850 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 msgr2=0x7f2dbc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00e9f0 tx=0x7f2dac00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 shutdown_connections 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f2da40a3dc0 0x7f2da40a6280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2dbc106f30 0x7f2dbc19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2dbc100e10 0x7f2dbc198c10 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 --2- 172.21.15.105:0/1825680022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2dbc100450 0x7f2dbc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.572+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 >> 172.21.15.105:0/1825680022 conn(0x7f2dbc0773e0 msgr2=0x7f2dbc104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.573+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 shutdown_connections 2024-04-22T06:39:31.574 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:31.573+0000 7f2dc0cfd640 1 -- 172.21.15.105:0/1825680022 wait complete. 2024-04-22T06:39:31.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:active 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:active 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:active 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:active 2024-04-22T06:39:31.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:31 smithi112 ceph-mon[31236]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:active 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:active 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:active 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:active 2024-04-22T06:39:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:31 smithi177 ceph-mon[31346]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/85174468,v1:172.21.15.105:6841/85174468] up:active 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/678281724,v1:172.21.15.177:6837/678281724] up:active 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3064875873,v1:172.21.15.177:6839/3064875873] up:active 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/1826997924,v1:172.21.15.112:6837/1826997924] up:active 2024-04-22T06:39:32.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:31 smithi105 ceph-mon[27916]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:39:32.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:32.793+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:32.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:32.793+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:39:32.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:32.793+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:39:32.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:32.793+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:39:32.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:32.793+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-22T06:39:32.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:32 smithi112 ceph-mon[31236]: pgmap v68: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-04-22T06:39:32.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:39:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:39:32.953+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:39:32.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:39:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:39:32.953+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:39:32.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:39:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:39:32.953+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-22T06:39:32.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:39:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:39:32.953+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-22T06:39:32.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:32 smithi177 ceph-mon[31346]: pgmap v68: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-04-22T06:39:33.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:32 smithi105 ceph-mon[27916]: pgmap v68: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.1 MiB/s wr, 17 op/s 2024-04-22T06:39:34.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:34 smithi177 ceph-mon[31346]: pgmap v69: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 70 op/s 2024-04-22T06:39:35.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:34 smithi105 ceph-mon[27916]: pgmap v69: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 70 op/s 2024-04-22T06:39:35.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:34 smithi112 ceph-mon[31236]: pgmap v69: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.6 MiB/s wr, 70 op/s 2024-04-22T06:39:36.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:36 smithi177 ceph-mon[31346]: pgmap v70: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-04-22T06:39:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:36 smithi105 ceph-mon[27916]: pgmap v70: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-04-22T06:39:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:36 smithi112 ceph-mon[31236]: pgmap v70: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 1.3 MiB/s wr, 55 op/s 2024-04-22T06:39:38.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:38 smithi177 ceph-mon[31346]: pgmap v71: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.0 MiB/s wr, 45 op/s 2024-04-22T06:39:39.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:38 smithi105 ceph-mon[27916]: pgmap v71: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.0 MiB/s wr, 45 op/s 2024-04-22T06:39:39.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:38 smithi112 ceph-mon[31236]: pgmap v71: 97 pgs: 97 active+clean; 2.6 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.0 MiB/s wr, 45 op/s 2024-04-22T06:39:40.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:40 smithi177 ceph-mon[31346]: pgmap v72: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-04-22T06:39:41.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:40 smithi105 ceph-mon[27916]: pgmap v72: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-04-22T06:39:41.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:40 smithi112 ceph-mon[31236]: pgmap v72: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 1.3 MiB/s wr, 44 op/s 2024-04-22T06:39:42.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:42 smithi177 ceph-mon[31346]: pgmap v73: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 867 KiB/s wr, 36 op/s 2024-04-22T06:39:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:42 smithi105 ceph-mon[27916]: pgmap v73: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 867 KiB/s wr, 36 op/s 2024-04-22T06:39:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:42 smithi112 ceph-mon[31236]: pgmap v73: 97 pgs: 97 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 867 KiB/s wr, 36 op/s 2024-04-22T06:39:43.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:43.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:43 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:43 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:39:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:43 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.665+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/3116834071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 msgr2=0x7ff9ec10df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.665+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/3116834071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec10df50 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009ca0 tx=0x7ff9e402c0f0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/3116834071 shutdown_connections 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/3116834071 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec10df50 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/3116834071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec102180 0x7ff9ec105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/3116834071 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 0x7ff9ec101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/3116834071 >> 172.21.15.105:0/3116834071 conn(0x7ff9ec0773e0 msgr2=0x7ff9ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/3116834071 shutdown_connections 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/3116834071 wait complete. 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 Processor -- start 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- start start 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 0x7ff9ec199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec19a200 0x7ff9ec1a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1112f0 con 0x7ff9ec106380 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec111170 con 0x7ff9ec19a200 2024-04-22T06:39:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.666+0000 7ff9f3ac1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec111470 con 0x7ff9ec1017d0 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36186/0 (socket says 172.21.15.105:36186) 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 -- 172.21.15.105:0/1863881721 learned_addr learned my addr 172.21.15.105:0/1863881721 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f205e640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec19a200 0x7ff9ec1a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f185d640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 0x7ff9ec199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 -- 172.21.15.105:0/1863881721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 msgr2=0x7ff9ec199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 0x7ff9ec199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 -- 172.21.15.105:0/1863881721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec19a200 msgr2=0x7ff9ec1a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec19a200 0x7ff9ec1a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 -- 172.21.15.105:0/1863881721 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec1a4f80 con 0x7ff9ec106380 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f105c640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00d770 tx=0x7ff9dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9e2ffd640 1 -- 172.21.15.105:0/1863881721 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc017070 con 0x7ff9ec106380 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ec1a5270 con 0x7ff9ec106380 2024-04-22T06:39:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9e2ffd640 1 -- 172.21.15.105:0/1863881721 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc009c30 con 0x7ff9ec106380 2024-04-22T06:39:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9e2ffd640 1 -- 172.21.15.105:0/1863881721 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc00f620 con 0x7ff9ec106380 2024-04-22T06:39:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.667+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ec1a5810 con 0x7ff9ec106380 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.670+0000 7ff9e2ffd640 1 -- 172.21.15.105:0/1863881721 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7ff9dc01a070 con 0x7ff9ec106380 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.670+0000 7ff9f6794740 1 Processor -- start 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 -- start start 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4b8d0 0x56284fe4a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 0x56284fe5a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56284fdaf8d0 con 0x56284fe584f0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56284fd4ce80 con 0x56284fe4ab60 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f6794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56284fdab890 con 0x56284fe4b8d0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9e2ffd640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 0x7ff9d80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f205e640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 0x7ff9d80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9e2ffd640 1 -- 172.21.15.105:0/1863881721 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7ff9dc0d0550 con 0x7ff9ec106380 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 0x56284fe5a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:48404/0 (socket says 172.21.15.105:48404) 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 -- 172.21.15.105:0/1088470043 learned_addr learned my addr 172.21.15.105:0/1088470043 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f205e640 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4b8d0 0x56284fe4a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 -- 172.21.15.105:0/1088470043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4b8d0 msgr2=0x56284fe4a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4b8d0 0x56284fe4a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 -- 172.21.15.105:0/1088470043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 msgr2=0x56284fe5a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 0x56284fe5a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.677 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f185d640 1 -- 172.21.15.105:0/1088470043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56284fd6e320 con 0x56284fe4ab60 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.671+0000 7ff9f105c640 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 0x56284fe5a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.672+0000 7ff9f205e640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 0x7ff9d80a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4004af0 tx=0x7ff9e40099f0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.672+0000 7ff9f185d640 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800cbe0 tx=0x7ff9e800e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f58fffb7acfdbf server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.672+0000 7ff9d7fff640 1 -- 172.21.15.105:0/1088470043 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8010430 con 0x56284fe4ab60 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.672+0000 7ff9d7fff640 1 -- 172.21.15.105:0/1088470043 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9e8010a80 con 0x56284fe4ab60 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.672+0000 7ff9d7fff640 1 -- 172.21.15.105:0/1088470043 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e801a9e0 con 0x56284fe4ab60 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- 172.21.15.105:0/1088470043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 msgr2=0x56284fe4afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800cbe0 tx=0x7ff9e800e350 comp rx=0 tx=0).stop 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- 172.21.15.105:0/1088470043 shutdown_connections 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe584f0 0x56284fe5a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4ab60 0x56284fe4afe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 --2- 172.21.15.105:0/1088470043 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4b8d0 0x56284fe4a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- 172.21.15.105:0/1088470043 >> 172.21.15.105:0/1088470043 conn(0x56284fe11c90 msgr2=0x56284fe45850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- 172.21.15.105:0/1088470043 shutdown_connections 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- 172.21.15.105:0/1088470043 wait complete. 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 -- start start 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.673+0000 7ff9f6794740 1 Processor -- start 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 0x56284fe55220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 0x56284fe557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56284fe5b220 con 0x56284fe4ab60 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56284fd37eb0 con 0x56284fe4b8d0 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56284fe5b3a0 con 0x56284fe584f0 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f6794740 10 client.0 _ll_register_callbacks cb 0x56284fd64b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f185d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 0x56284fe55220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f185d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36214/0 (socket says 172.21.15.105:36214) 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f185d640 1 -- 172.21.15.105:0/539919098 learned_addr learned my addr 172.21.15.105:0/539919098 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f205e640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 0x56284fe557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 -- 172.21.15.105:0/539919098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 msgr2=0x56284fe557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 0x56284fe557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 -- 172.21.15.105:0/539919098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 msgr2=0x56284fe57e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 -- 172.21.15.105:0/539919098 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56284fac2210 con 0x56284fe4b8d0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f205e640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 0x56284fe557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f185d640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9f105c640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 0x56284fe55220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0cda60 tx=0x7ff9dc09c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98a0bd6a5f8ea99 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.674+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0b5060 con 0x56284fe4b8d0 2024-04-22T06:39:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7ff9dc098920 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9e8010a80 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff9dc021040 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 10 client.25718 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56284fb8e620 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9e8010a80 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff9dc0bbbf0 con 0x56284fe4b8d0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 10 client.25718 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 20 client.25718 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fe68500 0x56284fe6a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 4 client.25718 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56284fdc3880 con 0x56284fe68500 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.675+0000 7ff9f185d640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fe68500 0x56284fe6a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9d57fa640 10 client.25718 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9f185d640 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fe68500 0x56284fe6a910 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=252bf6b23df30ee1 server_cookie=825d35a961694f1d in_seq=0 out_seq=0 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9d57fa640 1 -- 172.21.15.105:0/539919098 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56284fdc3880 con 0x56284fe68500 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9d57fa640 10 client.25718 handle_command_reply: tid=0 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9f6794740 1 client.25718 shutdown 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 msgr2=0x56284fe55220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.676+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 0x56284fe55220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0cda60 tx=0x7ff9dc09c9b0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 shutdown_connections 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fe68500 0x56284fe6a910 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe584f0 0x56284fe557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe4b8d0 0x56284fe55220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9d57fa640 0 client.25718 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 --2- 172.21.15.105:0/539919098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4ab60 0x56284fe57e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 >> 172.21.15.105:0/539919098 conn(0x56284fe11c90 msgr2=0x56284fe475a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 shutdown_connections 2024-04-22T06:39:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 1 -- 172.21.15.105:0/539919098 wait complete. 2024-04-22T06:39:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.677+0000 7ff9f6794740 20 client.25718 trim_cache size 0 max 16384 2024-04-22T06:39:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.691+0000 7ff9f6794740 1 Processor -- start 2024-04-22T06:39:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 -- start start 2024-04-22T06:39:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe64120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe57e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1abed0 con 0x56284fe4b8d0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1ada10 con 0x56284fe584f0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f6794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1ac5a0 con 0x56284fe4ab60 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36224/0 (socket says 172.21.15.105:36224) 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 -- 172.21.15.105:0/3060308543 learned_addr learned my addr 172.21.15.105:0/3060308543 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f205e640 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe57e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f185d640 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe64120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 -- 172.21.15.105:0/3060308543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 msgr2=0x56284fe64120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe64120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 -- 172.21.15.105:0/3060308543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 msgr2=0x56284fe57e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe57e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 -- 172.21.15.105:0/3060308543 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56284fe49e30 con 0x56284fe4b8d0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.692+0000 7ff9f105c640 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ce060 tx=0x7ff9dc010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42f9a85b1d8a6e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9d77fe640 1 -- 172.21.15.105:0/3060308543 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc07d020 con 0x56284fe4b8d0 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9d77fe640 1 -- 172.21.15.105:0/3060308543 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0a4f20 con 0x56284fe4b8d0 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9d77fe640 1 -- 172.21.15.105:0/3060308543 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0b5040 con 0x56284fe4b8d0 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 -- 172.21.15.105:0/3060308543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 msgr2=0x56284fe57950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ce060 tx=0x7ff9dc010c00 comp rx=0 tx=0).stop 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 -- 172.21.15.105:0/3060308543 shutdown_connections 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe57e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe57950 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 --2- 172.21.15.105:0/3060308543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe64120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 -- 172.21.15.105:0/3060308543 >> 172.21.15.105:0/3060308543 conn(0x56284fe11c90 msgr2=0x56284fe45380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.693+0000 7ff9f6794740 1 -- 172.21.15.105:0/3060308543 shutdown_connections 2024-04-22T06:39:44.694 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 -- 172.21.15.105:0/3060308543 wait complete. 2024-04-22T06:39:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 -- start start 2024-04-22T06:39:44.695 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 Processor -- start 2024-04-22T06:39:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe60b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe52c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.698 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1ad340 con 0x56284fe4b8d0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1acc70 con 0x56284fe584f0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff9ec1a9eb0 con 0x56284fe4ab60 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f6794740 10 client.0 _ll_register_callbacks cb 0x7ff9dc098620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f105c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36234/0 (socket says 172.21.15.105:36234) 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.694+0000 7ff9f105c640 1 -- 172.21.15.105:0/677848641 learned_addr learned my addr 172.21.15.105:0/677848641 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f205e640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe52c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f185d640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe60b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 -- 172.21.15.105:0/677848641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 msgr2=0x56284fe60b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe60b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 -- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 msgr2=0x56284fe52c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe52c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 -- 172.21.15.105:0/677848641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56284fe58ff0 con 0x56284fe4b8d0 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f205e640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe52c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:39:44.699 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f185d640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe60b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f105c640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc09c240 tx=0x7ff9dc09bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340ce27a239e9df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0a6020 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7ff9dc09b700 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9dc0a4f20 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff9dc07d030 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f6794740 10 client.16446 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56284fe58ff0 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.695+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc0a4f20 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ff9dc0a17f0 con 0x56284fe4b8d0 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 10 client.16446 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 20 client.16446 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fecee30 0x56284fed1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 4 client.16446 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56284fd4f3d0 con 0x56284fecee30 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f185d640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fecee30 0x56284fed1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9d57fa640 10 client.16446 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.696+0000 7ff9f185d640 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fecee30 0x56284fed1220 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20fb1c96085f286 server_cookie=51e2dec4c3948e8a in_seq=0 out_seq=0 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9d57fa640 1 -- 172.21.15.105:0/677848641 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56284fd4f3d0 con 0x56284fecee30 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9d57fa640 10 client.16446 handle_command_reply: tid=0 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 client.16446 shutdown 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 msgr2=0x56284fe52510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc09c240 tx=0x7ff9dc09bdf0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 shutdown_connections 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56284fecee30 0x56284fed1220 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56284fe584f0 0x56284fe52c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9d57fa640 0 client.16446 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:44.704 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56284fe4b8d0 0x56284fe52510 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 --2- 172.21.15.105:0/677848641 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56284fe4ab60 0x56284fe60b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 >> 172.21.15.105:0/677848641 conn(0x56284fe11c90 msgr2=0x56284fe536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 shutdown_connections 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 1 -- 172.21.15.105:0/677848641 wait complete. 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.705+0000 7ff9f6794740 20 client.16446 trim_cache size 0 max 16384 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:39:44.705 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:39:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 msgr2=0x7ff9d80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 0x7ff9d80a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4004af0 tx=0x7ff9e40099f0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 msgr2=0x7ff9ec199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:44.706 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00d770 tx=0x7ff9dc00dc30 comp rx=0 tx=0).stop 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 shutdown_connections 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff9d80a3dc0 0x7ff9d80a6280 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff9ec19a200 0x7ff9ec1a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9ec106380 0x7ff9ec199c70 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 --2- 172.21.15.105:0/1863881721 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff9ec1017d0 0x7ff9ec199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 >> 172.21.15.105:0/1863881721 conn(0x7ff9ec0773e0 msgr2=0x7ff9ec104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 shutdown_connections 2024-04-22T06:39:44.707 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:44.706+0000 7ff9f3ac1640 1 -- 172.21.15.105:0/1863881721 wait complete. 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:44 smithi177 ceph-mon[31346]: pgmap v74: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 16 MiB/s wr, 45 op/s 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:44 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:44 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:44.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:44 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:44 smithi105 ceph-mon[27916]: pgmap v74: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 16 MiB/s wr, 45 op/s 2024-04-22T06:39:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:44 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:44 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:44 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:44 smithi112 ceph-mon[31236]: pgmap v74: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 16 MiB/s wr, 45 op/s 2024-04-22T06:39:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:44 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:39:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:44 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:39:45.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:44 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:39:45.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:45.607+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:45.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:45.607+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:39:45.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:45.607+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:39:45.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:45.607+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:39:45.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:45.607+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:39:46.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:46 smithi177 ceph-mon[31346]: pgmap v75: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:47.164 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.1 2024-04-22T06:39:47.165 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:39:47.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:46 smithi105 ceph-mon[27916]: pgmap v75: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:47.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:46 smithi112 ceph-mon[31236]: pgmap v75: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1101813157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 msgr2=0x7fb9581025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1101813157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb9581025e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009900 tx=0x7fb94c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1101813157 shutdown_connections 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1101813157 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95810eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1101813157 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958102bb0 0x7fb958106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1101813157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb9581025e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1101813157 >> 172.21.15.105:0/1101813157 conn(0x7fb9580793e0 msgr2=0x7fb9580797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1101813157 shutdown_connections 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1101813157 wait complete. 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 Processor -- start 2024-04-22T06:39:47.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- start start 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958198610 0x7fb958198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb958112290 con 0x7fb958102200 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb958112110 con 0x7fb958198610 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.274+0000 7fb95f0d2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb958112410 con 0x7fb9581072d0 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36240/0 (socket says 172.21.15.105:36240) 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 -- 172.21.15.105:0/1440623390 learned_addr learned my addr 172.21.15.105:0/1440623390 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95d66f640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958198610 0x7fb958198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb957fff640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 -- 172.21.15.105:0/1440623390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 msgr2=0x7fb95819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 -- 172.21.15.105:0/1440623390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958198610 msgr2=0x7fb958198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958198610 0x7fb958198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 -- 172.21.15.105:0/1440623390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb958199130 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb957fff640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95ce6e640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005940 tx=0x7fb94c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c043020 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9581993c0 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb94c02cbe0 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.275+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c003bf0 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.276+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9581a5610 con 0x7fb958102200 2024-04-22T06:39:47.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.277+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb958068550 con 0x7fb958102200 2024-04-22T06:39:47.278 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.278+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb94c046020 con 0x7fb958102200 2024-04-22T06:39:47.278 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.278+0000 7fb955ffb640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 0x7fb9480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.278 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.279+0000 7fb957fff640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 0x7fb9480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.278 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.279+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7fb94c0f2bb0 con 0x7fb958102200 2024-04-22T06:39:47.278 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.279+0000 7fb957fff640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 0x7fb9480a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb958101870 tx=0x7fb944005e50 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.283+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb94c04d020 con 0x7fb958102200 2024-04-22T06:39:47.554 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.556+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb958199b40 con 0x7fb958102200 2024-04-22T06:39:47.555 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.557+0000 7fb955ffb640 1 -- 172.21.15.105:0/1440623390 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11950 (secure 0 0 0) 0x7fb94c0bf120 con 0x7fb958102200 2024-04-22T06:39:47.555 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:39:47.555 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:39:31.042056+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":25432,"mds_2":25426,"mds_3":25415,"mds_4":16080},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":41,"state":"up:active","state_seq":17,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16398":{"gid":16398,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/798740274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":798740274},{"type":"v1","addr":"172.21.15.105:6839","nonce":798740274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":39,"state":"up:active","state_seq":17,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":45,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":43,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25685":{"gid":25685,"name":"e","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/3361835468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":3361835468},{"type":"v1","addr":"172.21.15.112:6833","nonce":3361835468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25690":{"gid":25690,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/3918665016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3918665016},{"type":"v1","addr":"172.21.15.177:6833","nonce":3918665016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1127626841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1127626841},{"type":"v1","addr":"172.21.15.112:6835","nonce":1127626841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,16080,25432,25426,25415]},"id":1}]} 2024-04-22T06:39:47.556 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 49 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 msgr2=0x7fb9480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 0x7fb9480a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb958101870 tx=0x7fb944005e50 comp rx=0 tx=0).stop 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 msgr2=0x7fb95819e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb94c005940 tx=0x7fb94c02c7e0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 shutdown_connections 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb9480a3d20 0x7fb9480a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb958198610 0x7fb958198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb9581072d0 0x7fb95819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 --2- 172.21.15.105:0/1440623390 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb958102200 0x7fb95819e460 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.559 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 >> 172.21.15.105:0/1440623390 conn(0x7fb9580793e0 msgr2=0x7fb9580fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.559 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 shutdown_connections 2024-04-22T06:39:47.559 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.559+0000 7fb95f0d2640 1 -- 172.21.15.105:0/1440623390 wait complete. 2024-04-22T06:39:47.573 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db / --timeout 60 --expiration 81.1 --await' 2024-04-22T06:39:47.573 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db / --timeout 60 --expiration 81.1 --await 2024-04-22T06:39:47.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 -- 172.21.15.105:0/3003587324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8106160 msgr2=0x7f51a810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 --2- 172.21.15.105:0/3003587324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8106160 0x7f51a810dd30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009ca0 tx=0x7f51a002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 -- 172.21.15.105:0/3003587324 shutdown_connections 2024-04-22T06:39:47.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 --2- 172.21.15.105:0/3003587324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8106160 0x7f51a810dd30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.715 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 --2- 172.21.15.105:0/3003587324 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8101a40 0x7f51a8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 --2- 172.21.15.105:0/3003587324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a8101090 0x7f51a8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 -- 172.21.15.105:0/3003587324 >> 172.21.15.105:0/3003587324 conn(0x7f51a80771d0 msgr2=0x7f51a80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 -- 172.21.15.105:0/3003587324 shutdown_connections 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.714+0000 7f51afb23640 1 -- 172.21.15.105:0/3003587324 wait complete. 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 Processor -- start 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 -- start start 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8106160 0x7f51a8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a819bef0 0x7f51a819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81110d0 con 0x7f51a8101090 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51a8110f50 con 0x7f51a819bef0 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51afb23640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51a8111250 con 0x7f51a8106160 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51ad8bf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51ad8bf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36248/0 (socket says 172.21.15.105:36248) 2024-04-22T06:39:47.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51ad8bf640 1 -- 172.21.15.105:0/243660265 learned_addr learned my addr 172.21.15.105:0/243660265 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51ad0be640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8106160 0x7f51a8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.715+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a819bef0 0x7f51a819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 -- 172.21.15.105:0/243660265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8106160 msgr2=0x7f51a8197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8106160 0x7f51a8197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 -- 172.21.15.105:0/243660265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a819bef0 msgr2=0x7f51a819c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a819bef0 0x7f51a819c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 -- 172.21.15.105:0/243660265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a819c9e0 con 0x7f51a8101090 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e9f0 tx=0x7f519c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f519affd640 1 -- 172.21.15.105:0/243660265 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c00cd80 con 0x7f51a8101090 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f519affd640 1 -- 172.21.15.105:0/243660265 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f519c0108e0 con 0x7f51a8101090 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a819ccd0 con 0x7f51a8101090 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f519affd640 1 -- 172.21.15.105:0/243660265 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c018980 con 0x7f51a8101090 2024-04-22T06:39:47.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.716+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a806a8d0 con 0x7f51a8101090 2024-04-22T06:39:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.718+0000 7f519affd640 1 -- 172.21.15.105:0/243660265 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f519c010430 con 0x7f51a8101090 2024-04-22T06:39:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.718+0000 7f51b27ef740 1 Processor -- start 2024-04-22T06:39:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 -- start start 2024-04-22T06:39:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e122e0 0x563096e1a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e1d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563096d720b0 con 0x563096e119c0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563096d23680 con 0x563096e1aec0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51b27ef740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563096d2fc60 con 0x563096e122e0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ae0c0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ae0c0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36250/0 (socket says 172.21.15.105:36250) 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ae0c0640 1 -- 172.21.15.105:0/378709701 learned_addr learned my addr 172.21.15.105:0/378709701 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f519affd640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 0x7f51940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e1d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e122e0 0x563096e1a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 0x7f51940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 -- 172.21.15.105:0/378709701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e122e0 msgr2=0x563096e1a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e122e0 0x563096e1a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 -- 172.21.15.105:0/378709701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 msgr2=0x563096e11da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f51ad0be640 1 -- 172.21.15.105:0/378709701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563096b54c90 con 0x563096e1aec0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.719+0000 7f519affd640 1 -- 172.21.15.105:0/243660265 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f519c0d0d20 con 0x7f51a8101090 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 0x7f51940a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009a60 tx=0x7f51a002c0f0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f51ad0be640 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e1d2b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51a806a3c0 tx=0x7f51a4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdcc1acaeeb90344 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f5193fff640 1 -- 172.21.15.105:0/378709701 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004050 con 0x563096e1aec0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f5193fff640 1 -- 172.21.15.105:0/378709701 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a40054d0 con 0x563096e1aec0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.720+0000 7f5193fff640 1 -- 172.21.15.105:0/378709701 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004c60 con 0x563096e1aec0 2024-04-22T06:39:47.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- 172.21.15.105:0/378709701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 msgr2=0x563096e1d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e1d2b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51a806a3c0 tx=0x7f51a4011810 comp rx=0 tx=0).stop 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- 172.21.15.105:0/378709701 shutdown_connections 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e1d2b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e122e0 0x563096e1a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 --2- 172.21.15.105:0/378709701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e11da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- 172.21.15.105:0/378709701 >> 172.21.15.105:0/378709701 conn(0x563096e0c0f0 msgr2=0x563096e0c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- 172.21.15.105:0/378709701 shutdown_connections 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- 172.21.15.105:0/378709701 wait complete. 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 -- start start 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.721+0000 7f51b27ef740 1 Processor -- start 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 0x563096e17ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e18430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e18970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563096d75400 con 0x563096e122e0 2024-04-22T06:39:47.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x563096d31840 con 0x563096e119c0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x563096d7a720 con 0x563096e1aec0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51b27ef740 10 client.0 _ll_register_callbacks cb 0x563096d82ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e18430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e18430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36252/0 (socket says 172.21.15.105:36252) 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ad0be640 1 -- 172.21.15.105:0/4265665313 learned_addr learned my addr 172.21.15.105:0/4265665313 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e18970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 0x563096e17ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 -- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 msgr2=0x563096e17ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 0x563096e17ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 -- 172.21.15.105:0/4265665313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 msgr2=0x563096e18430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e18430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.722+0000 7f51ae0c0640 1 -- 172.21.15.105:0/4265665313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563096bb5420 con 0x563096e1aec0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 0x563096e17ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e18970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51a0005cc0 tx=0x7f51a00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efaa9ac01484c379 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-22T06:39:47.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0044050 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51a40054d0 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f51a0004040 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f51a0004240 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 10 client.25738 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563096a48580 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a40054d0 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f51a0034a40 con 0x563096e1aec0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 10 client.25738 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 10 client.25738 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.723+0000 7f51b27ef740 20 client.25738 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e2abd0 0x563096e2cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51b27ef740 4 client.25738 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563096d83b10 con 0x563096e2abd0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e2abd0 0x563096e2cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51917fa640 10 client.25738 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:47.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.724+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e2abd0 0x563096e2cfe0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f267736701f8db8e server_cookie=f51d32ab4b6235a3 in_seq=0 out_seq=0 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51917fa640 1 -- 172.21.15.105:0/4265665313 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563096d83b10 con 0x563096e2abd0 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51917fa640 10 client.25738 handle_command_reply: tid=0 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 client.25738 shutdown 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 msgr2=0x563096e18970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e18970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51a0005cc0 tx=0x7f51a00095c0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 shutdown_connections 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e2abd0 0x563096e2cfe0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e18970 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e18430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51917fa640 0 client.25738 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4265665313 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e119c0 0x563096e17ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 >> 172.21.15.105:0/4265665313 conn(0x563096e0c0f0 msgr2=0x563096e0de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 shutdown_connections 2024-04-22T06:39:47.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 1 -- 172.21.15.105:0/4265665313 wait complete. 2024-04-22T06:39:47.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.725+0000 7f51b27ef740 20 client.25738 trim_cache size 0 max 16384 2024-04-22T06:39:47.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 Processor -- start 2024-04-22T06:39:47.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 -- start start 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e119c0 0x563096e1a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81adad0 con 0x563096e122e0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81ac660 con 0x563096e1aec0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51b27ef740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81acd30 con 0x563096e119c0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36254/0 (socket says 172.21.15.105:36254) 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 -- 172.21.15.105:0/4018934962 learned_addr learned my addr 172.21.15.105:0/4018934962 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e119c0 0x563096e1a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 -- 172.21.15.105:0/4018934962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e119c0 msgr2=0x563096e1a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e119c0 0x563096e1a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 -- 172.21.15.105:0/4018934962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 msgr2=0x563096e328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e328f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.744+0000 7f51ad0be640 1 -- 172.21.15.105:0/4018934962 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563096e1d2b0 con 0x563096e122e0 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51ad0be640 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009710 tx=0x7f51a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee13f40b87aa342b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51937fe640 1 -- 172.21.15.105:0/4018934962 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4009e90 con 0x563096e122e0 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51937fe640 1 -- 172.21.15.105:0/4018934962 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a4028030 con 0x563096e122e0 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51937fe640 1 -- 172.21.15.105:0/4018934962 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4019430 con 0x563096e122e0 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51b27ef740 1 -- 172.21.15.105:0/4018934962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 msgr2=0x563096e268a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.745+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009710 tx=0x7f51a4009780 comp rx=0 tx=0).stop 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 -- 172.21.15.105:0/4018934962 shutdown_connections 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e1aec0 0x563096e328f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e122e0 0x563096e268a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 --2- 172.21.15.105:0/4018934962 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e119c0 0x563096e1a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 -- 172.21.15.105:0/4018934962 >> 172.21.15.105:0/4018934962 conn(0x563096e0c0f0 msgr2=0x563096e0b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 -- 172.21.15.105:0/4018934962 shutdown_connections 2024-04-22T06:39:47.750 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 -- 172.21.15.105:0/4018934962 wait complete. 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 -- start start 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.746+0000 7f51b27ef740 1 Processor -- start 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e90160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81a9f90 con 0x563096e119c0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81ad400 con 0x563096e122e0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f51a81aa5e0 con 0x563096e1aec0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51b27ef740 10 client.0 _ll_register_callbacks cb 0x7f51a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad8bf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:48480/0 (socket says 172.21.15.105:48480) 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad8bf640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36266/0 (socket says 172.21.15.105:36266) 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 -- 172.21.15.105:0/3861285220 learned_addr learned my addr 172.21.15.105:0/3861285220 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:47.751 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ae0c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e90160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 -- 172.21.15.105:0/3861285220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 msgr2=0x563096e90160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e90160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 -- 172.21.15.105:0/3861285220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 msgr2=0x563096e14f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 -- 172.21.15.105:0/3861285220 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563096e906a0 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ae0c0640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e90160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51ad0be640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009960 tx=0x7f51a40041b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f4448537ae3264 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.747+0000 7f51917fa640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a403b060 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51a4028030 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f51a40045b0 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f51a4042050 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 10 client.25748 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x563096e917c0 con 0x563096e122e0 2024-04-22T06:39:47.752 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a4028030 con 0x563096e122e0 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f51a4019690 con 0x563096e122e0 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 10 client.25748 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 10 client.25748 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 20 client.25748 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e91d10 0x563096e94130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 4 client.25748 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await": true}] 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.1, "await": true}) -- 0x563096d7d980 con 0x563096e91d10 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.748+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e91d10 0x563096e94130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.749+0000 7f51917fa640 10 client.25748 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:47.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.749+0000 7f51ad8bf640 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e91d10 0x563096e94130 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26be595d9ac15891 server_cookie=f98b84c50749020d in_seq=0 out_seq=0 2024-04-22T06:39:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51917fa640 1 -- 172.21.15.105:0/3861285220 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x563096d7d980 con 0x563096e91d10 2024-04-22T06:39:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51917fa640 10 client.25748 handle_command_reply: tid=0 2024-04-22T06:39:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 client.25748 shutdown 2024-04-22T06:39:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 msgr2=0x563096e15490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.758 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009960 tx=0x7f51a40041b0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 shutdown_connections 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x563096e91d10 0x563096e94130 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x563096e1aec0 0x563096e90160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x563096e122e0 0x563096e15490 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51917fa640 0 client.25748 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 --2- 172.21.15.105:0/3861285220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563096e119c0 0x563096e14f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 >> 172.21.15.105:0/3861285220 conn(0x563096e0c0f0 msgr2=0x563096e0b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 shutdown_connections 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.759+0000 7f51b27ef740 1 -- 172.21.15.105:0/3861285220 wait complete. 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51b27ef740 20 client.25748 trim_cache size 0 max 16384 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:39:47.759 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 49, 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 10, 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "e3efee66": { 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 10, 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:39:47.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 81.1, 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "QUIESCED", 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:47.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:39:47.762 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:39:47.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 msgr2=0x7f51940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 0x7f51940a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51a0009a60 tx=0x7f51a002c0f0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 msgr2=0x7f51a8197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f519c00e9f0 tx=0x7f519c00eeb0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 shutdown_connections 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f51940a3d20 0x7f51940a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51a819bef0 0x7f51a819c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f51a8106160 0x7f51a8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 --2- 172.21.15.105:0/243660265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51a8101090 0x7f51a8197630 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 >> 172.21.15.105:0/243660265 conn(0x7f51a80771d0 msgr2=0x7f51a80fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 shutdown_connections 2024-04-22T06:39:47.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:39:47.760+0000 7f51afb23640 1 -- 172.21.15.105:0/243660265 wait complete. 2024-04-22T06:39:47.769 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: e3efee66, quiesce took 0.6 seconds. Will release after: 21.1 2024-04-22T06:39:47.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1440623390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:47.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1440623390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1440623390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:39:48.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:48 smithi177 ceph-mon[31346]: pgmap v76: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:48 smithi105 ceph-mon[27916]: pgmap v76: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:48 smithi112 ceph-mon[31236]: pgmap v76: 97 pgs: 97 active+clean; 175 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 15 op/s 2024-04-22T06:39:51.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:50 smithi105 ceph-mon[27916]: pgmap v77: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 31 MiB/s wr, 26 op/s 2024-04-22T06:39:51.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:50 smithi112 ceph-mon[31236]: pgmap v77: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 31 MiB/s wr, 26 op/s 2024-04-22T06:39:51.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:50 smithi177 ceph-mon[31346]: pgmap v77: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 31 MiB/s wr, 26 op/s 2024-04-22T06:39:53.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:52 smithi105 ceph-mon[27916]: pgmap v78: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-04-22T06:39:53.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:52 smithi112 ceph-mon[31236]: pgmap v78: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-04-22T06:39:53.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:52 smithi177 ceph-mon[31346]: pgmap v78: 97 pgs: 97 active+clean; 259 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-04-22T06:39:55.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:54 smithi105 ceph-mon[27916]: pgmap v79: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-04-22T06:39:55.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:54 smithi112 ceph-mon[31236]: pgmap v79: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-04-22T06:39:55.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:54 smithi177 ceph-mon[31346]: pgmap v79: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 30 MiB/s wr, 25 op/s 2024-04-22T06:39:57.042 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:56 smithi177 ceph-mon[31346]: pgmap v80: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:39:57.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:56 smithi105 ceph-mon[27916]: pgmap v80: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:39:57.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:56 smithi112 ceph-mon[31236]: pgmap v80: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:39:57.805 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 -- 172.21.15.105:0/4011269640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 msgr2=0x7f3eb010dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.805 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/4011269640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb010dca0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009cd0 tx=0x7f3ea802c0f0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 -- 172.21.15.105:0/4011269640 shutdown_connections 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/4011269640 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb010dca0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/4011269640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0101130 0x7f3eb01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/4011269640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0106f90 0x7f3eb0107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 -- 172.21.15.105:0/4011269640 >> 172.21.15.105:0/4011269640 conn(0x7f3eb0077c30 msgr2=0x7f3eb0078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 -- 172.21.15.105:0/4011269640 shutdown_connections 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.804+0000 7f3eb63e2640 1 -- 172.21.15.105:0/4011269640 wait complete. 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 Processor -- start 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 -- start start 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0101130 0x7f3eb0102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb01030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01110a0 con 0x7f3eb0101af0 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0110f20 con 0x7f3eb0101130 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb63e2640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0111220 con 0x7f3eb0106f90 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb01030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb497f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb497f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:39766/0 (socket says 172.21.15.105:39766) 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eb497f640 1 -- 172.21.15.105:0/1627077395 learned_addr learned my addr 172.21.15.105:0/1627077395 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.805+0000 7f3eaffff640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0101130 0x7f3eb0102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 -- 172.21.15.105:0/1627077395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0101130 msgr2=0x7f3eb0102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0101130 0x7f3eb0102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 -- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 msgr2=0x7f3eb01030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb01030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 -- 172.21.15.105:0/1627077395 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb01a6af0 con 0x7f3eb0106f90 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8033000 tx=0x7f3ea8034650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3ead7fa640 1 -- 172.21.15.105:0/1627077395 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8041020 con 0x7f3eb0106f90 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3ead7fa640 1 -- 172.21.15.105:0/1627077395 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea8032970 con 0x7f3eb0106f90 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb01a6d80 con 0x7f3eb0106f90 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3ead7fa640 1 -- 172.21.15.105:0/1627077395 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8039040 con 0x7f3eb0106f90 2024-04-22T06:39:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.806+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb01a7270 con 0x7f3eb0106f90 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.808+0000 7f3ead7fa640 1 -- 172.21.15.105:0/1627077395 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f3ea8044020 con 0x7f3eb0106f90 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 Processor -- start 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 -- start start 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3ead7fa640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 0x7f3e980a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 0x55f54a93b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 0x7f3e980a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f54a8948d0 con 0x55f54a9390e0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f54a831e80 con 0x55f54a9308d0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f54a890890 con 0x55f54a92fb60 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3ead7fa640 1 -- 172.21.15.105:0/1627077395 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f3ea80f2590 con 0x7f3eb0106f90 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eb497f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 0x55f54a93b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:39782/0 (socket says 172.21.15.105:39782) 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaf7fe640 1 -- 172.21.15.105:0/1996291609 learned_addr learned my addr 172.21.15.105:0/1996291609 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.809+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:49982/0 (socket says 172.21.15.105:49982) 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 -- 172.21.15.105:0/1996291609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 msgr2=0x55f54a92ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 -- 172.21.15.105:0/1996291609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 msgr2=0x55f54a92f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 -- 172.21.15.105:0/1996291609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54a853320 con 0x55f54a9390e0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eaffff640 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 0x7f3e980a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40139b0 tx=0x7f3ea4009b50 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 0x55f54a93b4d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80efb50 tx=0x7f3ea80ba700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c909f16e35e6d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3e967fc640 1 -- 172.21.15.105:0/1996291609 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80348f0 con 0x55f54a9390e0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3e967fc640 1 -- 172.21.15.105:0/1996291609 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea80098d0 con 0x55f54a9390e0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.810+0000 7f3e967fc640 1 -- 172.21.15.105:0/1996291609 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80c7010 con 0x55f54a9390e0 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1996291609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 msgr2=0x55f54a93b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 0x55f54a93b4d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80efb50 tx=0x7f3ea80ba700 comp rx=0 tx=0).stop 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1996291609 shutdown_connections 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9390e0 0x55f54a93b4d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a92ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1996291609 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a92f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1996291609 >> 172.21.15.105:0/1996291609 conn(0x55f54a8f6c90 msgr2=0x55f54a92a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1996291609 shutdown_connections 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1996291609 wait complete. 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 -- start start 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.811+0000 7f3eb8fcc740 1 Processor -- start 2024-04-22T06:39:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a9389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9390e0 0x55f54a936290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f54a93bbd0 con 0x55f54a9308d0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f54a81ceb0 con 0x55f54a9390e0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55f54a93bd50 con 0x55f54a92fb60 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 10 client.0 _ll_register_callbacks cb 0x55f54a849b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42712/0 (socket says 172.21.15.105:42712) 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a9389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 -- 172.21.15.105:0/1861124508 learned_addr learned my addr 172.21.15.105:0/1861124508 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb497f640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9390e0 0x55f54a936290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 -- 172.21.15.105:0/1861124508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 msgr2=0x55f54a9389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a9389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 -- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9390e0 msgr2=0x55f54a936290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9390e0 0x55f54a936290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 -- 172.21.15.105:0/1861124508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54a5a7210 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400fab0 tx=0x7f3ea4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9199ed755b04255c server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3e8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4026060 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f3ea40035f0 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f54a8a8880 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3ea402d050 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.812+0000 7f3eb8fcc740 10 client.16482 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f54a673620 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f54a8a8880 con 0x55f54a9308d0 2024-04-22T06:39:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea4020050 con 0x55f54a9308d0 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 10 client.16482 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 10 client.16482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 20 client.16482 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a948d20 0x55f54a94b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 4 client.16482 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f54a8343d0 con 0x55f54a948d20 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.813+0000 7f3eaffff640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a948d20 0x55f54a94b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3eaffff640 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a948d20 0x55f54a94b110 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2759681b201e9477 server_cookie=5dcbd7af6c4c8e0a in_seq=0 out_seq=0 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3e8f7fe640 10 client.16482 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/1861124508 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f54a8343d0 con 0x55f54a948d20 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3e8f7fe640 10 client.16482 handle_command_reply: tid=0 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3eb8fcc740 1 client.16482 shutdown 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 msgr2=0x55f54a935d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.814+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400fab0 tx=0x7f3ea4004730 comp rx=0 tx=0).stop 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 shutdown_connections 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a948d20 0x55f54a94b110 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9390e0 0x55f54a936290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a9308d0 0x55f54a935d50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/1861124508 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a92fb60 0x55f54a9389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3e8f7fe640 0 client.16482 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 >> 172.21.15.105:0/1861124508 conn(0x55f54a8f6c90 msgr2=0x55f54a92c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 shutdown_connections 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/1861124508 wait complete. 2024-04-22T06:39:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.815+0000 7f3eb8fcc740 20 client.16482 trim_cache size 0 max 16384 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.829+0000 7f3eb8fcc740 1 Processor -- start 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.829+0000 7f3eb8fcc740 1 -- start start 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a950a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b01e0 con 0x55f54a92fb60 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b1d20 con 0x55f54a9308d0 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b08b0 con 0x55f54a9390e0 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42720/0 (socket says 172.21.15.105:42720) 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 -- 172.21.15.105:0/382966907 learned_addr learned my addr 172.21.15.105:0/382966907 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:57.833 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eb497f640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a950a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 -- 172.21.15.105:0/382966907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 msgr2=0x55f54a950a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a950a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 -- 172.21.15.105:0/382966907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 msgr2=0x55f54a9389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 -- 172.21.15.105:0/382966907 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54a94b770 con 0x55f54a92fb60 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3eaffff640 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55f54a94b9a0 tx=0x7f3ea0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c6d815d40f2b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3e95ffb640 1 -- 172.21.15.105:0/382966907 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004220 con 0x55f54a92fb60 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3e95ffb640 1 -- 172.21.15.105:0/382966907 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00043c0 con 0x55f54a92fb60 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.830+0000 7f3e95ffb640 1 -- 172.21.15.105:0/382966907 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0020920 con 0x55f54a92fb60 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/382966907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 msgr2=0x55f54a938480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x55f54a94b9a0 tx=0x7f3ea0005f90 comp rx=0 tx=0).stop 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/382966907 shutdown_connections 2024-04-22T06:39:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a950a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/382966907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a938480 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/382966907 >> 172.21.15.105:0/382966907 conn(0x55f54a8f6c90 msgr2=0x55f54a92a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/382966907 shutdown_connections 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/382966907 wait complete. 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 -- start start 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.831+0000 7f3eb8fcc740 1 Processor -- start 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a9336f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b1650 con 0x55f54a92fb60 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b0f80 con 0x55f54a9308d0 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01ae1c0 con 0x55f54a9390e0 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 10 client.0 _ll_register_callbacks cb 0x7f3ea40098b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:39:57.835 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42734/0 (socket says 172.21.15.105:42734) 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 -- 172.21.15.105:0/3527370062 learned_addr learned my addr 172.21.15.105:0/3527370062 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb497f640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a9336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 -- 172.21.15.105:0/3527370062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 msgr2=0x55f54a9336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a9336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 -- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 msgr2=0x55f54a9331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 -- 172.21.15.105:0/3527370062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54a92ee30 con 0x55f54a92fb60 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eaffff640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0005890 tx=0x7f3ea0005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f166c1fb5d16bdb server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3e8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:39:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea002d070 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f3ea00208a0 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.832+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea00043c0 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f3ea0039050 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 10 client.16488 fetch_fsmap learned FSMap version 49 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f54a9af500 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00043c0 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea0003a70 con 0x55f54a92fb60 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 10 client.16488 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 10 client.16488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 20 client.16488 populate_metadata read hostname 'smithi105' 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a9afc20 0x55f54a9b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:39:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 4 client.16488 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:39:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f54a8a22f0 con 0x55f54a9afc20 2024-04-22T06:39:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.833+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a9afc20 0x55f54a9b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:39:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.834+0000 7f3e8f7fe640 10 client.16488 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:57.838 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.834+0000 7f3eaf7fe640 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a9afc20 0x55f54a9b2040 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668af858c142c482 server_cookie=fd77114ab24836cf in_seq=0 out_seq=0 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3e8f7fe640 1 -- 172.21.15.105:0/3527370062 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f54a8a22f0 con 0x55f54a9afc20 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3e8f7fe640 10 client.16488 handle_command_reply: tid=0 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 client.16488 shutdown 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 msgr2=0x55f54a932c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0005890 tx=0x7f3ea0005760 comp rx=0 tx=0).stop 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 shutdown_connections 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55f54a9afc20 0x55f54a9b2040 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55f54a9390e0 0x55f54a9336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f54a9308d0 0x55f54a9331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 --2- 172.21.15.105:0/3527370062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f54a92fb60 0x55f54a932c70 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3e8f7fe640 0 client.16488 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 >> 172.21.15.105:0/3527370062 conn(0x55f54a8f6c90 msgr2=0x55f54a929040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 shutdown_connections 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 1 -- 172.21.15.105:0/3527370062 wait complete. 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.845+0000 7f3eb8fcc740 20 client.16488 trim_cache size 0 max 16384 2024-04-22T06:39:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:39:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:39:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:39:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 msgr2=0x7f3e980a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 0x7f3e980a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40139b0 tx=0x7f3ea4009b50 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 msgr2=0x7f3eb01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8033000 tx=0x7f3ea8034650 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 shutdown_connections 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f3e980a3dc0 0x7f3e980a6280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3eb0106f90 0x7f3eb01035e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3eb0101af0 0x7f3eb01030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 --2- 172.21.15.105:0/1627077395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3eb0101130 0x7f3eb0102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 >> 172.21.15.105:0/1627077395 conn(0x7f3eb0077c30 msgr2=0x7f3eb01003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 shutdown_connections 2024-04-22T06:39:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:39:57.846+0000 7f3eb63e2640 1 -- 172.21.15.105:0/1627077395 wait complete. 2024-04-22T06:39:59.011 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:39:58 smithi112 ceph-mon[31236]: pgmap v81: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:39:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:39:58 smithi105 ceph-mon[27916]: pgmap v81: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:39:59.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:39:58 smithi177 ceph-mon[31346]: pgmap v81: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 MiB/s wr, 16 op/s 2024-04-22T06:40:00.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:00.623+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:00.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:00.623+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:40:00.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:00.623+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:40:00.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:00.623+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:40:00.622 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:00.623+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:40:01.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:00 smithi105 ceph-mon[27916]: pgmap v82: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 23 op/s 2024-04-22T06:40:01.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:00 smithi105 ceph-mon[27916]: overall HEALTH_OK 2024-04-22T06:40:01.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:00 smithi112 ceph-mon[31236]: pgmap v82: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 23 op/s 2024-04-22T06:40:01.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:00 smithi112 ceph-mon[31236]: overall HEALTH_OK 2024-04-22T06:40:01.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:00 smithi177 ceph-mon[31346]: pgmap v82: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 23 op/s 2024-04-22T06:40:01.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:00 smithi177 ceph-mon[31346]: overall HEALTH_OK 2024-04-22T06:40:03.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:02 smithi105 ceph-mon[27916]: pgmap v83: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-04-22T06:40:03.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:02 smithi112 ceph-mon[31236]: pgmap v83: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-04-22T06:40:03.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:02 smithi177 ceph-mon[31346]: pgmap v83: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-04-22T06:40:05.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:04 smithi105 ceph-mon[27916]: pgmap v84: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-22T06:40:05.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:04 smithi112 ceph-mon[31236]: pgmap v84: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-22T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:04 smithi177 ceph-mon[31346]: pgmap v84: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-22T06:40:07.050 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:06 smithi177 ceph-mon[31346]: pgmap v85: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:07.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:06 smithi105 ceph-mon[27916]: pgmap v85: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:07.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:06 smithi112 ceph-mon[31236]: pgmap v85: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:08.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- 172.21.15.105:0/19970955 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 msgr2=0x7f7dc410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/19970955 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc410ede0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009900 tx=0x7f7dbc02bc30 comp rx=0 tx=0).stop 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- 172.21.15.105:0/19970955 shutdown_connections 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/19970955 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc410ede0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/19970955 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 0x7f7dc4069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/19970955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc4107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- 172.21.15.105:0/19970955 >> 172.21.15.105:0/19970955 conn(0x7f7dc4077e30 msgr2=0x7f7dc4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- 172.21.15.105:0/19970955 shutdown_connections 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- 172.21.15.105:0/19970955 wait complete. 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 Processor -- start 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- start start 2024-04-22T06:40:08.988 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 0x7f7dc4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4112170 con 0x7f7dc4069aa0 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4111ff0 con 0x7f7dc4106f50 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.986+0000 7f7dcae6a640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41122f0 con 0x7f7dc40690e0 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60180/0 (socket says 172.21.15.105:60180) 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 -- 172.21.15.105:0/2285885369 learned_addr learned my addr 172.21.15.105:0/2285885369 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc8c06640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 0x7f7dc4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc9407640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 -- 172.21.15.105:0/2285885369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 msgr2=0x7f7dc4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 0x7f7dc4198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 -- 172.21.15.105:0/2285885369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 msgr2=0x7f7dc419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:08.989 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 -- 172.21.15.105:0/2285885369 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc419d5d0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc9407640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc3fff640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7db400cce0 tx=0x7f7db400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4004b20 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7db4004cc0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc419d8c0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db400f7c0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.987+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc41a5150 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.988+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dc4068de0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.989+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f7db4005560 con 0x7f7dc4069aa0 2024-04-22T06:40:08.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.990+0000 7f7dc1ffb640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 0x7f7db00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:08.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.990+0000 7f7dc8c06640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 0x7f7db00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:08.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.990+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f7db40d0ac0 con 0x7f7dc4069aa0 2024-04-22T06:40:08.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.990+0000 7f7dc8c06640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 0x7f7db00a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0058c0 tx=0x7f7dac00a2d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:08.992 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:08.994+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7db4014030 con 0x7f7dc4069aa0 2024-04-22T06:40:09.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:08 smithi105 ceph-mon[27916]: pgmap v86: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:09.208 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.209+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dc4199930 con 0x7f7dc4069aa0 2024-04-22T06:40:09.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:08 smithi112 ceph-mon[31236]: pgmap v86: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:09.209 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.210+0000 7f7dc1ffb640 1 -- 172.21.15.105:0/2285885369 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11950 (secure 0 0 0) 0x7f7db409cf20 con 0x7f7dc4069aa0 2024-04-22T06:40:09.209 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:40:09.209 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:39:31.042056+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":25432,"mds_2":25426,"mds_3":25415,"mds_4":16080},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":41,"state":"up:active","state_seq":17,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16398":{"gid":16398,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/798740274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":798740274},{"type":"v1","addr":"172.21.15.105:6839","nonce":798740274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":39,"state":"up:active","state_seq":17,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":45,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":43,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25685":{"gid":25685,"name":"e","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/3361835468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":3361835468},{"type":"v1","addr":"172.21.15.112:6833","nonce":3361835468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25690":{"gid":25690,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/3918665016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3918665016},{"type":"v1","addr":"172.21.15.177:6833","nonce":3918665016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1127626841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1127626841},{"type":"v1","addr":"172.21.15.112:6835","nonce":1127626841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,16080,25432,25426,25415]},"id":1}]} 2024-04-22T06:40:09.209 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 49 2024-04-22T06:40:09.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 msgr2=0x7f7db00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 0x7f7db00a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7dac0058c0 tx=0x7f7dac00a2d0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 msgr2=0x7f7dc4198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7db400cce0 tx=0x7f7db400a620 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 shutdown_connections 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7db00a3d20 0x7f7db00a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7dc4106f50 0x7f7dc419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7dc4069aa0 0x7f7dc4198b70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 --2- 172.21.15.105:0/2285885369 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7dc40690e0 0x7f7dc4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 >> 172.21.15.105:0/2285885369 conn(0x7f7dc4077e30 msgr2=0x7f7dc40fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 shutdown_connections 2024-04-22T06:40:09.212 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.213+0000 7f7dcae6a640 1 -- 172.21.15.105:0/2285885369 wait complete. 2024-04-22T06:40:09.222 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --set-id e3efee66 --release --await' 2024-04-22T06:40:09.222 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db --set-id e3efee66 --release --await 2024-04-22T06:40:09.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:08 smithi177 ceph-mon[31346]: pgmap v86: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 25 op/s 2024-04-22T06:40:09.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 -- 172.21.15.105:0/4172977742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 msgr2=0x7feae806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.329 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 --2- 172.21.15.105:0/4172977742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae806ac20 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fead0009900 tx=0x7fead002bc30 comp rx=0 tx=0).stop 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 -- 172.21.15.105:0/4172977742 shutdown_connections 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 --2- 172.21.15.105:0/4172977742 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feae8107270 0x7feae810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 --2- 172.21.15.105:0/4172977742 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae806b1f0 0x7feae8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 --2- 172.21.15.105:0/4172977742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae806ac20 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 -- 172.21.15.105:0/4172977742 >> 172.21.15.105:0/4172977742 conn(0x7feae8077eb0 msgr2=0x7feae80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 -- 172.21.15.105:0/4172977742 shutdown_connections 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.329+0000 7feaee952640 1 -- 172.21.15.105:0/4172977742 wait complete. 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 Processor -- start 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 -- start start 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feae8107270 0x7feae8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae819cff0 0x7feae819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feae81121b0 con 0x7feae806a840 2024-04-22T06:40:09.330 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feae8112030 con 0x7feae819cff0 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaee952640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feae8112330 con 0x7feae8107270 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60198/0 (socket says 172.21.15.105:60198) 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 -- 172.21.15.105:0/502921248 learned_addr learned my addr 172.21.15.105:0/502921248 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feaeceef640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae819cff0 0x7feae819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 -- 172.21.15.105:0/502921248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feae8107270 msgr2=0x7feae8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feae8107270 0x7feae8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 -- 172.21.15.105:0/502921248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae819cff0 msgr2=0x7feae819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae819cff0 0x7feae819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.330+0000 7feae7fff640 1 -- 172.21.15.105:0/502921248 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae819dae0 con 0x7feae806a840 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feae7fff640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fead0005940 tx=0x7fead002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feae57fa640 1 -- 172.21.15.105:0/502921248 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0043020 con 0x7feae806a840 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae819dd70 con 0x7feae806a840 2024-04-22T06:40:09.331 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feae57fa640 1 -- 172.21.15.105:0/502921248 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead002cbe0 con 0x7feae806a840 2024-04-22T06:40:09.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feae57fa640 1 -- 172.21.15.105:0/502921248 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0003bf0 con 0x7feae806a840 2024-04-22T06:40:09.332 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.331+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae81a55c0 con 0x7feae806a840 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.333+0000 7feae57fa640 1 -- 172.21.15.105:0/502921248 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fead0046020 con 0x7feae806a840 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.333+0000 7feaf1624740 1 Processor -- start 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.333+0000 7feaf1624740 1 -- start start 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.333+0000 7feae57fa640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 0x7fead40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f7492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 0x7fead40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f6a5500 con 0x55ee7f749810 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f64b6a0 con 0x55ee7f740310 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaf1624740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f6a14c0 con 0x55ee7f740c30 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feae57fa640 1 -- 172.21.15.105:0/502921248 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7fead00f2c50 con 0x7feae806a840 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60210/0 (socket says 172.21.15.105:60210) 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 -- 172.21.15.105:0/4183620190 learned_addr learned my addr 172.21.15.105:0/4183620190 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feae77fe640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f7492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feae7fff640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 -- 172.21.15.105:0/4183620190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 msgr2=0x55ee7f7492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f7492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 -- 172.21.15.105:0/4183620190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 msgr2=0x55ee7f7406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.339 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 -- 172.21.15.105:0/4183620190 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee7f66a5e0 con 0x55ee7f749810 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feae7fff640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 0x7fead40a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feae000afd0 tx=0x7feae000ace0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feaeceef640 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feae0010c70 tx=0x7feae001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb8c07161e4851b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feade7fc640 1 -- 172.21.15.105:0/4183620190 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0023070 con 0x55ee7f749810 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feade7fc640 1 -- 172.21.15.105:0/4183620190 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae0003e20 con 0x55ee7f749810 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.334+0000 7feade7fc640 1 -- 172.21.15.105:0/4183620190 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0004630 con 0x55ee7f749810 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- 172.21.15.105:0/4183620190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 msgr2=0x55ee7f74bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7feae0010c70 tx=0x7feae001a000 comp rx=0 tx=0).stop 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- 172.21.15.105:0/4183620190 shutdown_connections 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f74bc00 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f7492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 --2- 172.21.15.105:0/4183620190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.340 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- 172.21.15.105:0/4183620190 >> 172.21.15.105:0/4183620190 conn(0x55ee7f73aa40 msgr2=0x55ee7f73ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- 172.21.15.105:0/4183620190 shutdown_connections 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- 172.21.15.105:0/4183620190 wait complete. 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.335+0000 7feaf1624740 1 -- start start 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 Processor -- start 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7466a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f746be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f74c5a0 con 0x55ee7f749810 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f65ea60 con 0x55ee7f740310 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x55ee7f74c720 con 0x55ee7f740c30 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 10 client.0 _ll_register_callbacks cb 0x55ee7f3b7110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60220/0 (socket says 172.21.15.105:60220) 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 -- 172.21.15.105:0/317043526 learned_addr learned my addr 172.21.15.105:0/317043526 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:09.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feae77fe640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f746be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feae7fff640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 -- 172.21.15.105:0/317043526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 msgr2=0x55ee7f746be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f746be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 -- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 msgr2=0x55ee7f7466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 -- 172.21.15.105:0/317043526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee7f3b7920 con 0x55ee7f749810 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feae7fff640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaeceef640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feae0004170 tx=0x7feae000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9285fdd16b9ff4e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feac77fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0034060 con 0x55ee7f749810 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7feae0003b80 con 0x55ee7f749810 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.336+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feae0003e20 con 0x55ee7f749810 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7feae003b050 con 0x55ee7f749810 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 10 client.16506 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:09.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ee7f483d30 con 0x55ee7f749810 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae0003e20 con 0x55ee7f749810 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7feae000f040 con 0x55ee7f749810 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 10 client.16506 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 10 client.16506 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 20 client.16506 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7598a0 0x55ee7f75bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 4 client.16506 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee7f678a10 con 0x55ee7f7598a0 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.337+0000 7feae7fff640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7598a0 0x55ee7f75bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feac77fe640 10 client.16506 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feae7fff640 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7598a0 0x55ee7f75bc90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d4587676a9f6b7 server_cookie=4d498a8879273ad5 in_seq=0 out_seq=0 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feac77fe640 1 -- 172.21.15.105:0/317043526 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee7f678a10 con 0x55ee7f7598a0 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feac77fe640 10 client.16506 handle_command_reply: tid=0 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feaf1624740 1 client.16506 shutdown 2024-04-22T06:40:09.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 msgr2=0x55ee7f747120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.338+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feae0004170 tx=0x7feae000eea0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 shutdown_connections 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7598a0 0x55ee7f75bc90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f749810 0x55ee7f747120 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feac77fe640 0 client.16506 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740c30 0x55ee7f746be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 --2- 172.21.15.105:0/317043526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 >> 172.21.15.105:0/317043526 conn(0x55ee7f73aa40 msgr2=0x55ee7f73c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 shutdown_connections 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 1 -- 172.21.15.105:0/317043526 wait complete. 2024-04-22T06:40:09.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.339+0000 7feaf1624740 20 client.16506 trim_cache size 0 max 16384 2024-04-22T06:40:09.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 Processor -- start 2024-04-22T06:40:09.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 -- start start 2024-04-22T06:40:09.358 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7554b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f749810 0x55ee7f7619e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feae81ae180 con 0x55ee7f740c30 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feae81afcc0 con 0x55ee7f740310 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feaf1624740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feae81ae850 con 0x55ee7f749810 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feae77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feae77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60234/0 (socket says 172.21.15.105:60234) 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.355+0000 7feae77fe640 1 -- 172.21.15.105:0/647296356 learned_addr learned my addr 172.21.15.105:0/647296356 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae7fff640 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feaeceef640 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f749810 0x55ee7f7619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 -- 172.21.15.105:0/647296356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f749810 msgr2=0x55ee7f7619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f749810 0x55ee7f7619e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 -- 172.21.15.105:0/647296356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 msgr2=0x55ee7f7554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.359 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 -- 172.21.15.105:0/647296356 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee7f742260 con 0x55ee7f740c30 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feae77fe640 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7feae806a420 tx=0x7fead800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388d4738a4497ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feaddffb640 1 -- 172.21.15.105:0/647296356 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8005220 con 0x55ee7f740c30 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feaddffb640 1 -- 172.21.15.105:0/647296356 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead8004c50 con 0x55ee7f740c30 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.356+0000 7feaddffb640 1 -- 172.21.15.105:0/647296356 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8010a30 con 0x55ee7f740c30 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- 172.21.15.105:0/647296356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 msgr2=0x55ee7f7614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7feae806a420 tx=0x7fead800bf50 comp rx=0 tx=0).stop 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- 172.21.15.105:0/647296356 shutdown_connections 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f749810 0x55ee7f7619e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7614a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- 172.21.15.105:0/647296356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f740310 0x55ee7f7554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- 172.21.15.105:0/647296356 >> 172.21.15.105:0/647296356 conn(0x55ee7f73aa40 msgr2=0x55ee7f73bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- 172.21.15.105:0/647296356 shutdown_connections 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- 172.21.15.105:0/647296356 wait complete. 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- start start 2024-04-22T06:40:09.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 Processor -- start 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740310 0x55ee7f7bd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 0x55ee7f74dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feae81af5f0 con 0x55ee7f740c30 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feae81aef20 con 0x55ee7f749810 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7feae81ac180 con 0x55ee7f740310 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feaf1624740 10 client.0 _ll_register_callbacks cb 0x7feae0010bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feae77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feae77fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60240/0 (socket says 172.21.15.105:60240) 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.357+0000 7feae77fe640 1 -- 172.21.15.105:0/2308540003 learned_addr learned my addr 172.21.15.105:0/2308540003 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feaeceef640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 0x55ee7f74dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae7fff640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740310 0x55ee7f7bd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 -- 172.21.15.105:0/2308540003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740310 msgr2=0x55ee7f7bd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740310 0x55ee7f7bd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 -- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 msgr2=0x55ee7f74dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 0x55ee7f74dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 -- 172.21.15.105:0/2308540003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee7f74e2c0 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feaeceef640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 0x55ee7f74dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feae77fe640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fead801a000 tx=0x7fead8004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc4f8287beed125 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8028020 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7fead8004240 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead8004c50 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fead8033050 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feaf1624740 10 client.16512 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ee7f7c4c30 con 0x55ee7f740c30 2024-04-22T06:40:09.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead8004c50 con 0x55ee7f740c30 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.358+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fead8019070 con 0x55ee7f740c30 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 10 client.16512 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 10 client.16512 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 20 client.16512 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7c5320 0x55ee7f7c7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 4 client.16512 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e3efee66", "release": true, "await": true}] 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e3efee66", "release": true, "await": true}) -- 0x55ee7f6b2f20 con 0x55ee7f7c5320 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feae7fff640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7c5320 0x55ee7f7c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feae7fff640 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7c5320 0x55ee7f7c7740 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b803ad7392d507d4 server_cookie=dfd30cc9b2908678 in_seq=0 out_seq=0 2024-04-22T06:40:09.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.359+0000 7feac77fe640 10 client.16512 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:09.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.361+0000 7feac77fe640 1 -- 172.21.15.105:0/2308540003 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ee7f6b2f20 con 0x55ee7f7c5320 2024-04-22T06:40:09.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.361+0000 7feac77fe640 10 client.16512 handle_command_reply: tid=0 2024-04-22T06:40:09.364 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 client.16512 shutdown 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 msgr2=0x55ee7f7bd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fead801a000 tx=0x7fead8004a50 comp rx=0 tx=0).stop 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 shutdown_connections 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x55ee7f7c5320 0x55ee7f7c7740 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feac77fe640 0 client.16512 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ee7f749810 0x55ee7f74dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ee7f740c30 0x55ee7f7bd9e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 --2- 172.21.15.105:0/2308540003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55ee7f740310 0x55ee7f7bd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 >> 172.21.15.105:0/2308540003 conn(0x55ee7f73aa40 msgr2=0x55ee7f745480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 shutdown_connections 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.362+0000 7feaf1624740 1 -- 172.21.15.105:0/2308540003 wait complete. 2024-04-22T06:40:09.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.363+0000 7feaf1624740 20 client.16512 trim_cache size 0 max 16384 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 49, 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 12, 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": { 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "e3efee66": { 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "version": 12, 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "age_ref": 0.0, 2024-04-22T06:40:09.366 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "timeout": 60.0, 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "expiration": 81.1, 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "members": { 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "file:/": { 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "excluded": false, 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "state": { 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RELEASED", 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:40:09.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:40:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:40:09.368 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:40:09.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.363+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 msgr2=0x7fead40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.363+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 0x7fead40a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feae000afd0 tx=0x7feae000ace0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.368 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.363+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 msgr2=0x7feae8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.363+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fead0005940 tx=0x7fead002c7e0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 shutdown_connections 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fead40a3d20 0x7fead40a61e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feae819cff0 0x7feae819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7feae8107270 0x7feae8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 --2- 172.21.15.105:0/502921248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae806a840 0x7feae8198730 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 >> 172.21.15.105:0/502921248 conn(0x7feae8077eb0 msgr2=0x7feae8108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 shutdown_connections 2024-04-22T06:40:09.369 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:09.364+0000 7feaee952640 1 -- 172.21.15.105:0/502921248 wait complete. 2024-04-22T06:40:09.375 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: e3efee66, seconds elapsed: 22.2 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f01e1d93700 time 2024-04-22T06:40:09.361101+0000 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01f53ce3dd] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.627 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: clone() 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.360+0000 7f01e1d93700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f01e1d93700 time 2024-04-22T06:40:09.361101+0000 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01f53ce3dd] 2024-04-22T06:40:09.628 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: clone() 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: *** Caught signal (Aborted) ** 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: in thread 7f01e1d93700 thread_name:quiesce.agt 2024-04-22T06:40:09.629 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01f4156d20] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: gsignal() 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: abort() 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01f53ce437] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 13: clone() 2024-04-22T06:40:09.630 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.362+0000 7f01e1d93700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: in thread 7f01e1d93700 thread_name:quiesce.agt 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01f4156d20] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: gsignal() 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: abort() 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01f53ce437] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.631 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 13: clone() 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: -2> 2024-04-22T06:40:09.360+0000 7f01e1d93700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f01e1d93700 time 2024-04-22T06:40:09.361101+0000 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01f53ce3dd] 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: clone() 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 0> 2024-04-22T06:40:09.362+0000 7f01e1d93700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: in thread 7f01e1d93700 thread_name:quiesce.agt 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.633 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01f4156d20] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: gsignal() 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: abort() 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01f53ce437] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 13: clone() 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.634 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: -2> 2024-04-22T06:40:09.360+0000 7f01e1d93700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f01e1d93700 time 2024-04-22T06:40:09.361101+0000 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f01f53ce3dd] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: clone() 2024-04-22T06:40:09.635 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 0> 2024-04-22T06:40:09.362+0000 7f01e1d93700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: in thread 7f01e1d93700 thread_name:quiesce.agt 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01f4156d20] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2: gsignal() 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 3: abort() 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f01f53ce437] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f01f53ce5a3] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x560f48cade9c] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x560f48cadf9f] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x560f48bdc0d3] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x560f48bdc03c] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x560f48e5bd3f] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x560f48ec4560] 2024-04-22T06:40:09.636 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f01f414c1ca] 2024-04-22T06:40:09.637 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 13: clone() 2024-04-22T06:40:09.637 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.637 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64050]: 2024-04-22T06:40:09.637 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:09 smithi112 podman[64748]: 2024-04-22 06:40:09.416429989 +0000 UTC m=+0.022529025 container died d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f282cdbf700 time 2024-04-22T06:40:09.361052+0000 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28403fa3dd] 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: clone() 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.361+0000 7f282cdbf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f282cdbf700 time 2024-04-22T06:40:09.361052+0000 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28403fa3dd] 2024-04-22T06:40:09.669 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.670 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.670 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.670 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: clone() 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: *** Caught signal (Aborted) ** 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: in thread 7f282cdbf700 thread_name:quiesce.agt 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f283f182d20] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: gsignal() 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: abort() 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28403fa437] 2024-04-22T06:40:09.671 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 13: clone() 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.363+0000 7f282cdbf700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: in thread 7f282cdbf700 thread_name:quiesce.agt 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f283f182d20] 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: gsignal() 2024-04-22T06:40:09.672 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: abort() 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28403fa437] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 13: clone() 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: -2> 2024-04-22T06:40:09.361+0000 7f282cdbf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f282cdbf700 time 2024-04-22T06:40:09.361052+0000 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.673 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28403fa3dd] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: clone() 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 0> 2024-04-22T06:40:09.363+0000 7f282cdbf700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.674 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: in thread 7f282cdbf700 thread_name:quiesce.agt 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f283f182d20] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: gsignal() 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: abort() 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28403fa437] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.675 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 13: clone() 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: -2> 2024-04-22T06:40:09.361+0000 7f282cdbf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f282cdbf700 time 2024-04-22T06:40:09.361052+0000 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f28403fa3dd] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.676 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: clone() 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 0> 2024-04-22T06:40:09.363+0000 7f282cdbf700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: in thread 7f282cdbf700 thread_name:quiesce.agt 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f283f182d20] 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2: gsignal() 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 3: abort() 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f28403fa437] 2024-04-22T06:40:09.677 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f28403fa5a3] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x5573ef7b7e9c] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x5573ef7b7f9f] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x5573ef6e60d3] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x5573ef6e603c] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x5573ef965d3f] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x5573ef9ce560] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f283f1781ca] 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 13: clone() 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[85878]: 2024-04-22T06:40:09.678 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:09 smithi105 podman[88057]: 2024-04-22 06:40:09.415867401 +0000 UTC m=+0.023574697 container died 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:40:09.712 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7faf88b00700 time 2024-04-22T06:40:09.361249+0000 2024-04-22T06:40:09.712 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.712 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.712 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf9c13b3dd] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: clone() 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.361+0000 7faf88b00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7faf88b00700 time 2024-04-22T06:40:09.361249+0000 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.713 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf9c13b3dd] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7a6fe5e700 time 2024-04-22T06:40:09.361248+0000 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7a834993dd] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.714 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: clone() 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.361+0000 7f7a6fe5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7a6fe5e700 time 2024-04-22T06:40:09.361248+0000 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7a834993dd] 2024-04-22T06:40:09.715 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: clone() 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: *** Caught signal (Aborted) ** 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: in thread 7faf88b00700 thread_name:quiesce.agt 2024-04-22T06:40:09.716 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf9aec3d20] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: gsignal() 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: abort() 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf9c13b437] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 13: clone() 2024-04-22T06:40:09.717 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.363+0000 7faf88b00700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: in thread 7faf88b00700 thread_name:quiesce.agt 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: clone() 2024-04-22T06:40:09.718 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: *** Caught signal (Aborted) ** 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: in thread 7f7a6fe5e700 thread_name:quiesce.agt 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a82221d20] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: gsignal() 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: abort() 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7a83499437] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.719 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 13: clone() 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.363+0000 7f7a6fe5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: in thread 7f7a6fe5e700 thread_name:quiesce.agt 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf9aec3d20] 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: gsignal() 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: abort() 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf9c13b437] 2024-04-22T06:40:09.720 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 13: clone() 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: -1> 2024-04-22T06:40:09.361+0000 7faf88b00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7faf88b00700 time 2024-04-22T06:40:09.361249+0000 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.721 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf9c13b3dd] 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a82221d20] 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: gsignal() 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: abort() 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7a83499437] 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.722 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 13: clone() 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: -1> 2024-04-22T06:40:09.361+0000 7f7a6fe5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7a6fe5e700 time 2024-04-22T06:40:09.361248+0000 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.723 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7a834993dd] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: clone() 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 0> 2024-04-22T06:40:09.363+0000 7faf88b00700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.724 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: in thread 7faf88b00700 thread_name:quiesce.agt 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf9aec3d20] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: gsignal() 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: abort() 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf9c13b437] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.725 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 13: clone() 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: clone() 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.726 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 0> 2024-04-22T06:40:09.363+0000 7f7a6fe5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: in thread 7f7a6fe5e700 thread_name:quiesce.agt 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a82221d20] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: gsignal() 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: abort() 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7a83499437] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.727 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 13: clone() 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: -1> 2024-04-22T06:40:09.361+0000 7faf88b00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7faf88b00700 time 2024-04-22T06:40:09.361249+0000 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7faf9c13b3dd] 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.728 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: clone() 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 0> 2024-04-22T06:40:09.363+0000 7faf88b00700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: in thread 7faf88b00700 thread_name:quiesce.agt 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7faf9aec3d20] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2: gsignal() 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 3: abort() 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7faf9c13b437] 2024-04-22T06:40:09.729 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7faf9c13b5a3] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: -1> 2024-04-22T06:40:09.361+0000 7f7a6fe5e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: In function 'void Locker::_drop_locks(MutationImpl*, std::set*, bool)' thread 7f7a6fe5e700 time 2024-04-22T06:40:09.361248+0000 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3156-g86f7587a/rpm/el8/BUILD/ceph-19.0.0-3156-g86f7587a/src/mds/Locker.cc: 720: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7a834993dd] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.730 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: clone() 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 0> 2024-04-22T06:40:09.363+0000 7f7a6fe5e700 -1 *** Caught signal (Aborted) ** 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: in thread 7f7a6fe5e700 thread_name:quiesce.agt 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: ceph version 19.0.0-3156-g86f7587a (86f7587a5a09af35f0895e1a2d08527638fae697) squid (dev) 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7a82221d20] 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2: gsignal() 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 3: abort() 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7a83499437] 2024-04-22T06:40:09.731 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d15a3) [0x7f7a834995a3] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x563df1693e9c] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x563df1693f9f] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x563df15c20d3] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x563df15c203c] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x563df1841d3f] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x563df18aa560] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7faf9aeb91ca] 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 13: clone() 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[65143]: 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:09 smithi177 podman[65788]: 2024-04-22 06:40:09.424769836 +0000 UTC m=+0.028025331 container died dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:40:09.732 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 6: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0xa2c) [0x561d15633e9c] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 7: (Locker::drop_locks(MutationImpl*, std::set, std::allocator >*)+0x7f) [0x561d15633f9f] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 8: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x243) [0x561d155620d3] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 9: (MDCache::request_cleanup(boost::intrusive_ptr const&)+0x1ac) [0x561d1556203c] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 10: /usr/bin/ceph-mds(+0x60dd3f) [0x561d157e1d3f] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 11: (QuiesceAgent::agent_thread_main()+0x1680) [0x561d1584a560] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7a822171ca] 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 13: clone() 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[65149]: 2024-04-22T06:40:09.733 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:09 smithi177 podman[65787]: 2024-04-22 06:40:09.424769856 +0000 UTC m=+0.027699614 container died 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:40:09.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:09 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2285885369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:09.986 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:09 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2285885369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:09.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:09 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2285885369' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:10.409 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:10 smithi112 podman[64748]: 2024-04-22 06:40:10.154405337 +0000 UTC m=+0.760504360 container remove d81b022e138be885347d249802a252067e6971108c72f5fff51b5c9b0d5b3d9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-22T06:40:10.409 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:10 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:40:10.427 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:10 smithi105 podman[88057]: 2024-04-22 06:40:10.177841775 +0000 UTC m=+0.785549069 container remove 7be455d6ccdd1f8813ceffcba66f2fe6f79307c217798d52366b4457f0013ea7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-22T06:40:10.427 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:10 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:40:10.489 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:10 smithi177 podman[65787]: 2024-04-22 06:40:10.202197421 +0000 UTC m=+0.805127220 container remove 1452e149a7a8b30b07bae226ae1be81feae3488a42afd8e7b494bb85d9b9d75f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:40:10.490 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:10 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:40:10.708 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:10 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-22T06:40:10.747 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:10 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-22T06:40:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- 172.21.15.105:0/2981861470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d41010f0 msgr2=0x7f05d41014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 --2- 172.21.15.105:0/2981861470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d41010f0 0x7f05d41014d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009900 tx=0x7f05c402bc30 comp rx=0 tx=0).stop 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- 172.21.15.105:0/2981861470 shutdown_connections 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 --2- 172.21.15.105:0/2981861470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d4106170 0x7f05d410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 --2- 172.21.15.105:0/2981861470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4101aa0 0x7f05d4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 --2- 172.21.15.105:0/2981861470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d41010f0 0x7f05d41014d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- 172.21.15.105:0/2981861470 >> 172.21.15.105:0/2981861470 conn(0x7f05d4077fd0 msgr2=0x7f05d40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- 172.21.15.105:0/2981861470 shutdown_connections 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- 172.21.15.105:0/2981861470 wait complete. 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 Processor -- start 2024-04-22T06:40:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.943+0000 7f05da32b640 1 -- start start 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 0x7f05d4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 0x7f05d4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41121f0 con 0x7f05d419cf80 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4112070 con 0x7f05d41010f0 2024-04-22T06:40:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05da32b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05d4112370 con 0x7f05d4106170 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d88c8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 0x7f05d4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 0x7f05d4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d88c8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60256/0 (socket says 172.21.15.105:60256) 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d88c8640 1 -- 172.21.15.105:0/3870385750 learned_addr learned my addr 172.21.15.105:0/3870385750 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 -- 172.21.15.105:0/3870385750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 msgr2=0x7f05d4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 0x7f05d4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 -- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 msgr2=0x7f05d419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 -- 172.21.15.105:0/3870385750 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d419da70 con 0x7f05d4106170 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d3fff640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 0x7f05d4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d88c8640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.944+0000 7f05d37fe640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 0x7f05d4198c50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f05c800e9c0 tx=0x7f05c800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.945+0000 7f05d17fa640 1 -- 172.21.15.105:0/3870385750 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c800ccc0 con 0x7f05d4106170 2024-04-22T06:40:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.945+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d419dd60 con 0x7f05d4106170 2024-04-22T06:40:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.945+0000 7f05d17fa640 1 -- 172.21.15.105:0/3870385750 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80108e0 con 0x7f05d4106170 2024-04-22T06:40:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.945+0000 7f05d17fa640 1 -- 172.21.15.105:0/3870385750 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80187a0 con 0x7f05d4106170 2024-04-22T06:40:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.945+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d41a5550 con 0x7f05d4106170 2024-04-22T06:40:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.947+0000 7f05d17fa640 1 -- 172.21.15.105:0/3870385750 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f05c8010430 con 0x7f05d4106170 2024-04-22T06:40:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d17fa640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 0x7f05bc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 Processor -- start 2024-04-22T06:40:10.948 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 -- start start 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d3fff640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 0x7f05bc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d17fa640 1 -- 172.21.15.105:0/3870385750 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f05c80d0ec0 con 0x7f05d4106170 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03ccb60 0x557ac03ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 0x557ac03dc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557ac03318d0 con 0x557ac03cd8d0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ac02cee80 con 0x557ac03da4f0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05dcffb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ac032d890 con 0x557ac03ccb60 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60270/0 (socket says 172.21.15.105:60270) 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d3fff640 1 -- 172.21.15.105:0/1862427831 learned_addr learned my addr 172.21.15.105:0/1862427831 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 0x557ac03dc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03ccb60 0x557ac03ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 -- 172.21.15.105:0/1862427831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03ccb60 msgr2=0x557ac03ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03ccb60 0x557ac03ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 -- 172.21.15.105:0/1862427831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 msgr2=0x557ac03cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d88c8640 1 -- 172.21.15.105:0/1862427831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ac02f0320 con 0x557ac03da4f0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.948+0000 7f05d3fff640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.949+0000 7f05d3fff640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 0x7f05bc0a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f05c402c170 tx=0x7f05c4004370 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.949+0000 7f05d88c8640 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 0x557ac03dc8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05cc011760 tx=0x7f05cc011c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4719950aa4e3f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.949+0000 7f05ba7fc640 1 -- 172.21.15.105:0/1862427831 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc004d20 con 0x557ac03da4f0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.949+0000 7f05ba7fc640 1 -- 172.21.15.105:0/1862427831 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc00fe90 con 0x557ac03da4f0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.949+0000 7f05ba7fc640 1 -- 172.21.15.105:0/1862427831 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc019430 con 0x557ac03da4f0 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- 172.21.15.105:0/1862427831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 msgr2=0x557ac03dc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 0x557ac03dc8e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f05cc011760 tx=0x7f05cc011c20 comp rx=0 tx=0).stop 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- 172.21.15.105:0/1862427831 shutdown_connections 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03da4f0 0x557ac03dc8e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03ccb60 0x557ac03ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1862427831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- 172.21.15.105:0/1862427831 >> 172.21.15.105:0/1862427831 conn(0x557ac0393c90 msgr2=0x557ac03c7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- 172.21.15.105:0/1862427831 shutdown_connections 2024-04-22T06:40:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- 172.21.15.105:0/1862427831 wait complete. 2024-04-22T06:40:10.951 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.950+0000 7f05dcffb740 1 -- start start 2024-04-22T06:40:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 Processor -- start 2024-04-22T06:40:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03d9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 0x557ac03d7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 0x557ac03d7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557ac03dd220 con 0x557ac03da4f0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557ac02b9eb0 con 0x557ac03ccb60 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x557ac03dd3a0 con 0x557ac03cd8d0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05dcffb740 10 client.0 _ll_register_callbacks cb 0x557ac02e6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d88c8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 0x557ac03d7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d37fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 0x557ac03d7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03d9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d88c8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 0x557ac03d7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60278/0 (socket says 172.21.15.105:60278) 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d88c8640 1 -- 172.21.15.105:0/1865392646 learned_addr learned my addr 172.21.15.105:0/1865392646 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 -- 172.21.15.105:0/1865392646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 msgr2=0x557ac03d7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 0x557ac03d7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 -- 172.21.15.105:0/1865392646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 msgr2=0x557ac03d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 0x557ac03d7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d3fff640 1 -- 172.21.15.105:0/1865392646 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ac0044210 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.951+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 0x557ac03d7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05d3fff640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03d9d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009230 tx=0x7f05c402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efaa8ba88cc7602e server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4051030 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f05c403a040 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05cc00fe90 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f05c405e050 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05dcffb740 10 client.25781 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557ac0110620 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05cc00fe90 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f05c404c040 con 0x557ac03ccb60 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05dcffb740 10 client.25781 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.952+0000 7f05dcffb740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05dcffb740 20 client.25781 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac03ea3d0 0x557ac03ec7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05dcffb740 4 client.25781 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ac0345880 con 0x557ac03ea3d0 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac03ea3d0 0x557ac03ec7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05b37fe640 10 client.25781 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.953+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac03ea3d0 0x557ac03ec7c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0ea86c426c40351 server_cookie=3799670d92146ebb in_seq=0 out_seq=0 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05b37fe640 1 -- 172.21.15.105:0/1865392646 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ac0345880 con 0x557ac03ea3d0 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05b37fe640 10 client.25781 handle_command_reply: tid=0 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 client.25781 shutdown 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 msgr2=0x557ac03d9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03d9d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009230 tx=0x7f05c402bc30 comp rx=0 tx=0).stop 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 shutdown_connections 2024-04-22T06:40:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac03ea3d0 0x557ac03ec7c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03da4f0 0x557ac03d7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03cd8d0 0x557ac03d7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1865392646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03d9d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 >> 172.21.15.105:0/1865392646 conn(0x557ac0393c90 msgr2=0x557ac03c95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05b37fe640 0 client.25781 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 shutdown_connections 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 1 -- 172.21.15.105:0/1865392646 wait complete. 2024-04-22T06:40:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.954+0000 7f05dcffb740 20 client.25781 trim_cache size 0 max 16384 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 Processor -- start 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 -- start start 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03e6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03d9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41abf20 con 0x557ac03cd8d0 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41ada60 con 0x557ac03ccb60 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.971+0000 7f05dcffb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41ac5f0 con 0x557ac03da4f0 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60294/0 (socket says 172.21.15.105:60294) 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 -- 172.21.15.105:0/1940233014 learned_addr learned my addr 172.21.15.105:0/1940233014 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d88c8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03d9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 -- 172.21.15.105:0/1940233014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 msgr2=0x557ac03d9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03d9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 -- 172.21.15.105:0/1940233014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 msgr2=0x557ac03e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03e6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 -- 172.21.15.105:0/1940233014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ac03cbe30 con 0x557ac03cd8d0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d3fff640 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05d37fe640 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f05c80ce430 tx=0x7f05c800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6c631d780d1edc server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05b9ffb640 1 -- 172.21.15.105:0/1940233014 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80a0fd0 con 0x557ac03cd8d0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05b9ffb640 1 -- 172.21.15.105:0/1940233014 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80a5890 con 0x557ac03cd8d0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.972+0000 7f05b9ffb640 1 -- 172.21.15.105:0/1940233014 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c8099290 con 0x557ac03cd8d0 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- 172.21.15.105:0/1940233014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 msgr2=0x557ac03c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f05c80ce430 tx=0x7f05c800efb0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- 172.21.15.105:0/1940233014 shutdown_connections 2024-04-22T06:40:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03d9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03cd8d0 0x557ac03c93b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 --2- 172.21.15.105:0/1940233014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03ccb60 0x557ac03e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- 172.21.15.105:0/1940233014 >> 172.21.15.105:0/1940233014 conn(0x557ac0393c90 msgr2=0x557ac03c8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- 172.21.15.105:0/1940233014 shutdown_connections 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- 172.21.15.105:0/1940233014 wait complete. 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.973+0000 7f05dcffb740 1 -- start start 2024-04-22T06:40:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 Processor -- start 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 0x557ac03dde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03de470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41ad390 con 0x557ac03ccb60 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41accc0 con 0x557ac03cd8d0 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f05d41a9f20 con 0x557ac03da4f0 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05dcffb740 10 client.0 _ll_register_callbacks cb 0x7f05c4004170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60304/0 (socket says 172.21.15.105:60304) 2024-04-22T06:40:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 0x557ac03dde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d3fff640 1 -- 172.21.15.105:0/2931856079 learned_addr learned my addr 172.21.15.105:0/2931856079 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d88c8640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03de470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 -- 172.21.15.105:0/2931856079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 msgr2=0x557ac03de470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03de470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 -- 172.21.15.105:0/2931856079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 msgr2=0x557ac03e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 -- 172.21.15.105:0/2931856079 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ac03dec70 con 0x557ac03cd8d0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d3fff640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05d37fe640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 0x557ac03dde60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f05c80ce430 tx=0x7f05c809cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3a447685e43026 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.974+0000 7f05b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c809af20 con 0x557ac03cd8d0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f05c8099290 con 0x557ac03cd8d0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05c80a5890 con 0x557ac03cd8d0 2024-04-22T06:40:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f05c80bd0d0 con 0x557ac03cd8d0 2024-04-22T06:40:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 10 client.25787 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557ac0450a50 con 0x557ac03cd8d0 2024-04-22T06:40:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c80a5890 con 0x557ac03cd8d0 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f05c8018940 con 0x557ac03cd8d0 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 10 client.25787 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 10 client.25787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 20 client.25787 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac0451110 0x557ac0453530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 4 client.25787 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557ac02d13d0 con 0x557ac0451110 2024-04-22T06:40:10.981 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.975+0000 7f05d3fff640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac0451110 0x557ac0453530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.976+0000 7f05b37fe640 10 client.25787 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.976+0000 7f05d3fff640 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac0451110 0x557ac0453530 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=763f6ca422ef14f1 server_cookie=f7a66ef3c0e6beb2 in_seq=0 out_seq=0 2024-04-22T06:40:10.983 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05b37fe640 1 -- 172.21.15.105:0/2931856079 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557ac02d13d0 con 0x557ac0451110 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05b37fe640 10 client.25787 handle_command_reply: tid=0 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 client.25787 shutdown 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 msgr2=0x557ac03dde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 0x557ac03dde60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f05c80ce430 tx=0x7f05c809cce0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 shutdown_connections 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x557ac0451110 0x557ac0453530 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x557ac03da4f0 0x557ac03de470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557ac03cd8d0 0x557ac03dde60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05b37fe640 0 client.25787 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 --2- 172.21.15.105:0/2931856079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557ac03ccb60 0x557ac03e2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.983+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 >> 172.21.15.105:0/2931856079 conn(0x557ac0393c90 msgr2=0x557ac03c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 shutdown_connections 2024-04-22T06:40:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05dcffb740 1 -- 172.21.15.105:0/2931856079 wait complete. 2024-04-22T06:40:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05dcffb740 20 client.25787 trim_cache size 0 max 16384 2024-04-22T06:40:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:{ 2024-04-22T06:40:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "message": "", 2024-04-22T06:40:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout: "return_code": 0 2024-04-22T06:40:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stdout:} 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 msgr2=0x7f05bc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 0x7f05bc0a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f05c402c170 tx=0x7f05c4004370 comp rx=0 tx=0).stop 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 msgr2=0x7f05d4198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 0x7f05d4198c50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f05c800e9c0 tx=0x7f05c800ee80 comp rx=0 tx=0).stop 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 shutdown_connections 2024-04-22T06:40:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f05bc0a3dc0 0x7f05bc0a6280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05d419cf80 0x7f05d419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05d4106170 0x7f05d4198c50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 --2- 172.21.15.105:0/3870385750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f05d41010f0 0x7f05d4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.984+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 >> 172.21.15.105:0/3870385750 conn(0x7f05d4077fd0 msgr2=0x7f05d40fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.985+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 shutdown_connections 2024-04-22T06:40:10.987 INFO:teuthology.task.background_exec.ubuntu@smithi105.front.sepia.ceph.com.smithi105.stderr:2024-04-22T06:40:10.985+0000 7f05da32b640 1 -- 172.21.15.105:0/3870385750 wait complete. 2024-04-22T06:40:10.987 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:10 smithi177 ceph-mon[31346]: pgmap v87: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-04-22T06:40:10.988 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:10 smithi177 podman[65788]: 2024-04-22 06:40:10.812075961 +0000 UTC m=+1.415331455 container remove dd9b0b499e3530ad0e3b8121662152947f10ab631db68d678c9626f509c2496b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:40:10.988 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:10 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-22T06:40:11.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:10 smithi105 ceph-mon[27916]: pgmap v87: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-04-22T06:40:11.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:10 smithi112 ceph-mon[31236]: pgmap v87: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 KiB/s wr, 30 op/s 2024-04-22T06:40:11.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:11 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-22T06:40:11.490 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:11 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-22T06:40:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:12 smithi105 ceph-mon[27916]: pgmap v88: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:40:13.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:12 smithi112 ceph-mon[31236]: pgmap v88: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:40:13.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:12 smithi177 ceph-mon[31346]: pgmap v88: 97 pgs: 97 active+clean; 259 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-22T06:40:14.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:14 smithi105 ceph-mon[27916]: pgmap v89: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 275 op/s 2024-04-22T06:40:15.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:14 smithi112 ceph-mon[31236]: pgmap v89: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 275 op/s 2024-04-22T06:40:15.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:14 smithi177 ceph-mon[31346]: pgmap v89: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 275 op/s 2024-04-22T06:40:15.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:15.638+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:15.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:15.638+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:40:15.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:15.638+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:40:15.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:15.638+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:40:15.637 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:15.638+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:40:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:16 smithi105 ceph-mon[27916]: pgmap v90: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:17.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:16 smithi112 ceph-mon[31236]: pgmap v90: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:17.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:16 smithi177 ceph-mon[31346]: pgmap v90: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:setting up random write file 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:done setting up random write file 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:starting fsync run 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0076s 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0049s 2024-04-22T06:40:17.403 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0049s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0054s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0047s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0047s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0048s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0050s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0052s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0047s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0046s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0045s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0047s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0049s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0049s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0047s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0051s 2024-04-22T06:40:17.404 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0048s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0048s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0047s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0046s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0044s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0045s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0049s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0046s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0047s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0047s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0041s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0045s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0050s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0045s 2024-04-22T06:40:17.405 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 21.4320s fsync time: 0.0064s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0051s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0039s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0040s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0040s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0003s fsync time: 0.0044s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0047s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0044s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:write time: 0.0002s fsync time: 0.0041s 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stdout:run done 40 fsyncs total, killing random writer 2024-04-22T06:40:17.406 INFO:tasks.workunit.client.0.smithi105.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-22T06:40:17.407 INFO:tasks.workunit.client.0.smithi105.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-22T06:40:17.407 INFO:tasks.workunit.client.0.smithi105.stderr:+ whereis lsof 2024-04-22T06:40:17.446 INFO:tasks.workunit.client.0.smithi105.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-04-22T06:40:17.447 INFO:tasks.workunit.client.0.smithi105.stderr:+ lsof 2024-04-22T06:40:17.465 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/e3d4fe72944bc6806d2ddaaae1bd207e95fb0a750a029b88c0e9ff2f93f3f574/merged 2024-04-22T06:40:17.465 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/90e74cc0324e92505f87ed653d2ad92b6940aa3189a446dcd2c7afb4bf52df21/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/1420345a501e37e659e5b256065258da40f487c6c3bab81c33894e8c185a9325/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c49b8dac48f50d0d3f34cc9dab36d71db15ff5691418d888c52ba0b4572ddbfd/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/7078112d2025222ab3de9dfd850d3dc79b344eb904f5a9ea2b891bbc8ef1b858/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/2d1ca434682d174f7442d743470657940a6a0370b2c7e73955ed74bdab6cd807/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/b93f4c5cf360e59ca93e76ce969248cc1e1b663e69b0cb31246e736a71fd7ba5/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/766f268fda2690b6cb4b78a02755678d05635509702359d0ce3a9f4057270190/merged 2024-04-22T06:40:17.466 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.467 INFO:tasks.workunit.client.0.smithi105.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/b2c9a6abed0608c9bca6459a1c359835d5b495294df275268b88423642728ce2/merged 2024-04-22T06:40:17.467 INFO:tasks.workunit.client.0.smithi105.stderr: Output information may be incomplete. 2024-04-22T06:40:17.538 INFO:tasks.workunit.client.0.smithi105.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-04-22T06:40:17.538 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-04-22T06:40:17.538 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-04-22T06:40:17.538 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-04-22T06:40:17.538 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_gp 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-04-22T06:40:17.539 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_gp 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_gp 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_gp 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_par_g 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_par_g 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_par_g 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_par_g 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-04-22T06:40:17.540 INFO:tasks.workunit.client.0.smithi105.stdout:slub_flus 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:slub_flus 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:slub_flus 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:slub_flus 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:netns 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:netns 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-04-22T06:40:17.541 INFO:tasks.workunit.client.0.smithi105.stdout:netns 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:netns 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 8 root cwd unknown /proc/8/cwd (readlink: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 8 root rtd unknown /proc/8/root (readlink: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 8 root txt unknown /proc/8/exe (readlink: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 8 root NOFD /proc/8/fd (opendir: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:mm_percpu 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-04-22T06:40:17.542 INFO:tasks.workunit.client.0.smithi105.stdout:mm_percpu 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:mm_percpu 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:mm_percpu 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 11 root cwd unknown /proc/11/cwd (readlink: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 11 root rtd unknown /proc/11/root (readlink: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 11 root txt unknown /proc/11/exe (readlink: Permission denied) 2024-04-22T06:40:17.543 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 11 root NOFD /proc/11/fd (opendir: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-04-22T06:40:17.544 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-04-22T06:40:17.545 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:pr/ttyS1 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:pr/ttyS1 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-04-22T06:40:17.546 INFO:tasks.workunit.client.0.smithi105.stdout:pr/ttyS1 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:pr/ttyS1 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-04-22T06:40:17.547 INFO:tasks.workunit.client.0.smithi105.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-04-22T06:40:17.548 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/0 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/0 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/0 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/0 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/1 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-04-22T06:40:17.549 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/1 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/1 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/1 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 23 root cwd unknown /proc/23/cwd (readlink: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 23 root rtd unknown /proc/23/root (readlink: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 23 root txt unknown /proc/23/exe (readlink: Permission denied) 2024-04-22T06:40:17.550 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 23 root NOFD /proc/23/fd (opendir: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:migration 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:migration 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:migration 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:migration 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-04-22T06:40:17.551 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-04-22T06:40:17.552 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/2 28 root cwd unknown /proc/28/cwd (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/2 28 root rtd unknown /proc/28/root (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/2 28 root txt unknown /proc/28/exe (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/2 28 root NOFD /proc/28/fd (opendir: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-04-22T06:40:17.553 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:migration 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:migration 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:migration 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:migration 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-04-22T06:40:17.554 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 32 root cwd unknown /proc/32/cwd (readlink: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 32 root rtd unknown /proc/32/root (readlink: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 32 root txt unknown /proc/32/exe (readlink: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 32 root NOFD /proc/32/fd (opendir: Permission denied) 2024-04-22T06:40:17.555 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 33 root cwd unknown /proc/33/cwd (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 33 root rtd unknown /proc/33/root (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 33 root txt unknown /proc/33/exe (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 33 root NOFD /proc/33/fd (opendir: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-04-22T06:40:17.556 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-04-22T06:40:17.557 INFO:tasks.workunit.client.0.smithi105.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 38 root cwd unknown /proc/38/cwd (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 38 root rtd unknown /proc/38/root (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 38 root txt unknown /proc/38/exe (readlink: Permission denied) 2024-04-22T06:40:17.558 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 38 root NOFD /proc/38/fd (opendir: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/4 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/4 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-04-22T06:40:17.559 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/4 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/4 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:migration 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-04-22T06:40:17.560 INFO:tasks.workunit.client.0.smithi105.stdout:migration 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:migration 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:migration 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 43 root cwd unknown /proc/43/cwd (readlink: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 43 root rtd unknown /proc/43/root (readlink: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 43 root txt unknown /proc/43/exe (readlink: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 43 root NOFD /proc/43/fd (opendir: Permission denied) 2024-04-22T06:40:17.561 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/5 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/5 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-04-22T06:40:17.562 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/5 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/5 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-04-22T06:40:17.563 INFO:tasks.workunit.client.0.smithi105.stdout:migration 48 root cwd unknown /proc/48/cwd (readlink: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:migration 48 root rtd unknown /proc/48/root (readlink: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:migration 48 root txt unknown /proc/48/exe (readlink: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:migration 48 root NOFD /proc/48/fd (opendir: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-04-22T06:40:17.564 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/6 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-04-22T06:40:17.565 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/6 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/6 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/6 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 53 root cwd unknown /proc/53/cwd (readlink: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 53 root rtd unknown /proc/53/root (readlink: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 53 root txt unknown /proc/53/exe (readlink: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 53 root NOFD /proc/53/fd (opendir: Permission denied) 2024-04-22T06:40:17.566 INFO:tasks.workunit.client.0.smithi105.stdout:migration 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:migration 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:migration 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:migration 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-04-22T06:40:17.567 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 56 root cwd unknown /proc/56/cwd (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 56 root rtd unknown /proc/56/root (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 56 root txt unknown /proc/56/exe (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 56 root NOFD /proc/56/fd (opendir: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 57 root cwd unknown /proc/57/cwd (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 57 root rtd unknown /proc/57/root (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 57 root txt unknown /proc/57/exe (readlink: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 57 root NOFD /proc/57/fd (opendir: Permission denied) 2024-04-22T06:40:17.568 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/7 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/7 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/7 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:cpuhp/7 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 59 root cwd unknown /proc/59/cwd (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 59 root rtd unknown /proc/59/root (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 59 root txt unknown /proc/59/exe (readlink: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:idle_inje 59 root NOFD /proc/59/fd (opendir: Permission denied) 2024-04-22T06:40:17.569 INFO:tasks.workunit.client.0.smithi105.stdout:migration 60 root cwd unknown /proc/60/cwd (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:migration 60 root rtd unknown /proc/60/root (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:migration 60 root txt unknown /proc/60/exe (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:migration 60 root NOFD /proc/60/fd (opendir: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 61 root cwd unknown /proc/61/cwd (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 61 root rtd unknown /proc/61/root (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 61 root txt unknown /proc/61/exe (readlink: Permission denied) 2024-04-22T06:40:17.570 INFO:tasks.workunit.client.0.smithi105.stdout:ksoftirqd 61 root NOFD /proc/61/fd (opendir: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-04-22T06:40:17.571 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 68 root cwd unknown /proc/68/cwd (readlink: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 68 root rtd unknown /proc/68/root (readlink: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 68 root txt unknown /proc/68/exe (readlink: Permission denied) 2024-04-22T06:40:17.572 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 68 root NOFD /proc/68/fd (opendir: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:kdevtmpfs 71 root cwd unknown /proc/71/cwd (readlink: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:kdevtmpfs 71 root rtd unknown /proc/71/root (readlink: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:kdevtmpfs 71 root txt unknown /proc/71/exe (readlink: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:kdevtmpfs 71 root NOFD /proc/71/fd (opendir: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:inet_frag 72 root cwd unknown /proc/72/cwd (readlink: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:inet_frag 72 root rtd unknown /proc/72/root (readlink: Permission denied) 2024-04-22T06:40:17.573 INFO:tasks.workunit.client.0.smithi105.stdout:inet_frag 72 root txt unknown /proc/72/exe (readlink: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:inet_frag 72 root NOFD /proc/72/fd (opendir: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:kauditd 73 root cwd unknown /proc/73/cwd (readlink: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:kauditd 73 root rtd unknown /proc/73/root (readlink: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:kauditd 73 root txt unknown /proc/73/exe (readlink: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:kauditd 73 root NOFD /proc/73/fd (opendir: Permission denied) 2024-04-22T06:40:17.574 INFO:tasks.workunit.client.0.smithi105.stdout:khungtask 75 root cwd unknown /proc/75/cwd (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:khungtask 75 root rtd unknown /proc/75/root (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:khungtask 75 root txt unknown /proc/75/exe (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:khungtask 75 root NOFD /proc/75/fd (opendir: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:oom_reape 76 root cwd unknown /proc/76/cwd (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:oom_reape 76 root rtd unknown /proc/76/root (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:oom_reape 76 root txt unknown /proc/76/exe (readlink: Permission denied) 2024-04-22T06:40:17.575 INFO:tasks.workunit.client.0.smithi105.stdout:oom_reape 76 root NOFD /proc/76/fd (opendir: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:writeback 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:writeback 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:writeback 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:writeback 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:kcompactd 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:kcompactd 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:kcompactd 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:kcompactd 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:ksmd 79 root cwd unknown /proc/79/cwd (readlink: Permission denied) 2024-04-22T06:40:17.576 INFO:tasks.workunit.client.0.smithi105.stdout:ksmd 79 root rtd unknown /proc/79/root (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:ksmd 79 root txt unknown /proc/79/exe (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:ksmd 79 root NOFD /proc/79/fd (opendir: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:khugepage 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:khugepage 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:khugepage 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:khugepage 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:cryptd 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:cryptd 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:cryptd 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:cryptd 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:kintegrit 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:kintegrit 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:kintegrit 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:kintegrit 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-04-22T06:40:17.577 INFO:tasks.workunit.client.0.smithi105.stdout:kblockd 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:kblockd 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:kblockd 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:kblockd 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:blkcg_pun 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:blkcg_pun 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:blkcg_pun 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:blkcg_pun 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:tpm_dev_w 86 root cwd unknown /proc/86/cwd (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:tpm_dev_w 86 root rtd unknown /proc/86/root (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:tpm_dev_w 86 root txt unknown /proc/86/exe (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:tpm_dev_w 86 root NOFD /proc/86/fd (opendir: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:md 87 root cwd unknown /proc/87/cwd (readlink: Permission denied) 2024-04-22T06:40:17.578 INFO:tasks.workunit.client.0.smithi105.stdout:md 87 root rtd unknown /proc/87/root (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md 87 root txt unknown /proc/87/exe (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md 87 root NOFD /proc/87/fd (opendir: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md_bitmap 88 root cwd unknown /proc/88/cwd (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md_bitmap 88 root rtd unknown /proc/88/root (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md_bitmap 88 root txt unknown /proc/88/exe (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:md_bitmap 88 root NOFD /proc/88/fd (opendir: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:edac-poll 89 root cwd unknown /proc/89/cwd (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:edac-poll 89 root rtd unknown /proc/89/root (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:edac-poll 89 root txt unknown /proc/89/exe (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:edac-poll 89 root NOFD /proc/89/fd (opendir: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:watchdogd 90 root cwd unknown /proc/90/cwd (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:watchdogd 90 root rtd unknown /proc/90/root (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:watchdogd 90 root txt unknown /proc/90/exe (readlink: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:watchdogd 90 root NOFD /proc/90/fd (opendir: Permission denied) 2024-04-22T06:40:17.579 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 92 root cwd unknown /proc/92/cwd (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 92 root rtd unknown /proc/92/root (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 92 root txt unknown /proc/92/exe (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 92 root NOFD /proc/92/fd (opendir: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kswapd0 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kswapd0 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kswapd0 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kswapd0 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kthrotld 100 root cwd unknown /proc/100/cwd (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kthrotld 100 root rtd unknown /proc/100/root (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kthrotld 100 root txt unknown /proc/100/exe (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:kthrotld 100 root NOFD /proc/100/fd (opendir: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:acpi_ther 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:acpi_ther 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-04-22T06:40:17.580 INFO:tasks.workunit.client.0.smithi105.stdout:acpi_ther 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:acpi_ther 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kmpath_rd 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kmpath_rd 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kmpath_rd 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kmpath_rd 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kaluad 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kaluad 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kaluad 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kaluad 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:mld 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:mld 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:mld 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:mld 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-04-22T06:40:17.581 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:ipv6_addr 114 root cwd unknown /proc/114/cwd (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:ipv6_addr 114 root rtd unknown /proc/114/root (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:ipv6_addr 114 root txt unknown /proc/114/exe (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:ipv6_addr 114 root NOFD /proc/114/fd (opendir: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kstrp 123 root cwd unknown /proc/123/cwd (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kstrp 123 root rtd unknown /proc/123/root (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kstrp 123 root txt unknown /proc/123/exe (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kstrp 123 root NOFD /proc/123/fd (opendir: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 124 root cwd unknown /proc/124/cwd (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 124 root rtd unknown /proc/124/root (readlink: Permission denied) 2024-04-22T06:40:17.582 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 124 root txt unknown /proc/124/exe (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 124 root NOFD /proc/124/fd (opendir: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:zswap-shr 131 root cwd unknown /proc/131/cwd (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:zswap-shr 131 root rtd unknown /proc/131/root (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:zswap-shr 131 root txt unknown /proc/131/exe (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:zswap-shr 131 root NOFD /proc/131/fd (opendir: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 132 root cwd unknown /proc/132/cwd (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 132 root rtd unknown /proc/132/root (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 132 root txt unknown /proc/132/exe (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 132 root NOFD /proc/132/fd (opendir: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 239 root cwd unknown /proc/239/cwd (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 239 root rtd unknown /proc/239/root (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 239 root txt unknown /proc/239/exe (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 239 root NOFD /proc/239/fd (opendir: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 279 root cwd unknown /proc/279/cwd (readlink: Permission denied) 2024-04-22T06:40:17.583 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 279 root rtd unknown /proc/279/root (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 279 root txt unknown /proc/279/exe (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 279 root NOFD /proc/279/fd (opendir: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 281 root cwd unknown /proc/281/cwd (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 281 root rtd unknown /proc/281/root (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 281 root txt unknown /proc/281/exe (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 281 root NOFD /proc/281/fd (opendir: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 284 root cwd unknown /proc/284/cwd (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 284 root rtd unknown /proc/284/root (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 284 root txt unknown /proc/284/exe (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 284 root NOFD /proc/284/fd (opendir: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 288 root cwd unknown /proc/288/cwd (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 288 root rtd unknown /proc/288/root (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 288 root txt unknown /proc/288/exe (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 288 root NOFD /proc/288/fd (opendir: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 380 root cwd unknown /proc/380/cwd (readlink: Permission denied) 2024-04-22T06:40:17.584 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 380 root rtd unknown /proc/380/root (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 380 root txt unknown /proc/380/exe (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 380 root NOFD /proc/380/fd (opendir: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 391 root cwd unknown /proc/391/cwd (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 391 root rtd unknown /proc/391/root (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 391 root txt unknown /proc/391/exe (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 391 root NOFD /proc/391/fd (opendir: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-wq 430 root cwd unknown /proc/430/cwd (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-wq 430 root rtd unknown /proc/430/root (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-wq 430 root txt unknown /proc/430/exe (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-wq 430 root NOFD /proc/430/fd (opendir: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:ata_sff 431 root cwd unknown /proc/431/cwd (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:ata_sff 431 root rtd unknown /proc/431/root (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:ata_sff 431 root txt unknown /proc/431/exe (readlink: Permission denied) 2024-04-22T06:40:17.585 INFO:tasks.workunit.client.0.smithi105.stdout:ata_sff 431 root NOFD /proc/431/fd (opendir: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-rese 433 root cwd unknown /proc/433/cwd (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-rese 433 root rtd unknown /proc/433/root (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-rese 433 root txt unknown /proc/433/exe (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-rese 433 root NOFD /proc/433/fd (opendir: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-dele 435 root cwd unknown /proc/435/cwd (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-dele 435 root rtd unknown /proc/435/root (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-dele 435 root txt unknown /proc/435/exe (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-dele 435 root NOFD /proc/435/fd (opendir: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:ixgbe 439 root cwd unknown /proc/439/cwd (readlink: Permission denied) 2024-04-22T06:40:17.586 INFO:tasks.workunit.client.0.smithi105.stdout:ixgbe 439 root rtd unknown /proc/439/root (readlink: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:ixgbe 439 root txt unknown /proc/439/exe (readlink: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:ixgbe 439 root NOFD /proc/439/fd (opendir: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-auth 440 root cwd unknown /proc/440/cwd (readlink: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-auth 440 root rtd unknown /proc/440/root (readlink: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-auth 440 root txt unknown /proc/440/exe (readlink: Permission denied) 2024-04-22T06:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:nvme-auth 440 root NOFD /proc/440/fd (opendir: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_0 444 root cwd unknown /proc/444/cwd (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_0 444 root rtd unknown /proc/444/root (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_0 444 root txt unknown /proc/444/exe (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_0 444 root NOFD /proc/444/fd (opendir: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 445 root cwd unknown /proc/445/cwd (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 445 root rtd unknown /proc/445/root (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 445 root txt unknown /proc/445/exe (readlink: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 445 root NOFD /proc/445/fd (opendir: Permission denied) 2024-04-22T06:40:17.588 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_1 446 root cwd unknown /proc/446/cwd (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_1 446 root rtd unknown /proc/446/root (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_1 446 root txt unknown /proc/446/exe (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_1 446 root NOFD /proc/446/fd (opendir: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 447 root cwd unknown /proc/447/cwd (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 447 root rtd unknown /proc/447/root (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 447 root txt unknown /proc/447/exe (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 447 root NOFD /proc/447/fd (opendir: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_2 448 root cwd unknown /proc/448/cwd (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_2 448 root rtd unknown /proc/448/root (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_2 448 root txt unknown /proc/448/exe (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_2 448 root NOFD /proc/448/fd (opendir: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 449 root cwd unknown /proc/449/cwd (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 449 root rtd unknown /proc/449/root (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 449 root txt unknown /proc/449/exe (readlink: Permission denied) 2024-04-22T06:40:17.589 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 449 root NOFD /proc/449/fd (opendir: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_3 451 root cwd unknown /proc/451/cwd (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_3 451 root rtd unknown /proc/451/root (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_3 451 root txt unknown /proc/451/exe (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_3 451 root NOFD /proc/451/fd (opendir: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 452 root cwd unknown /proc/452/cwd (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 452 root rtd unknown /proc/452/root (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 452 root txt unknown /proc/452/exe (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 452 root NOFD /proc/452/fd (opendir: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_4 459 root cwd unknown /proc/459/cwd (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_4 459 root rtd unknown /proc/459/root (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_4 459 root txt unknown /proc/459/exe (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_4 459 root NOFD /proc/459/fd (opendir: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 460 root cwd unknown /proc/460/cwd (readlink: Permission denied) 2024-04-22T06:40:17.590 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 460 root rtd unknown /proc/460/root (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 460 root txt unknown /proc/460/exe (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 460 root NOFD /proc/460/fd (opendir: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_5 461 root cwd unknown /proc/461/cwd (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_5 461 root rtd unknown /proc/461/root (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_5 461 root txt unknown /proc/461/exe (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_5 461 root NOFD /proc/461/fd (opendir: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 462 root cwd unknown /proc/462/cwd (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 462 root rtd unknown /proc/462/root (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 462 root txt unknown /proc/462/exe (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 462 root NOFD /proc/462/fd (opendir: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_6 463 root cwd unknown /proc/463/cwd (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_6 463 root rtd unknown /proc/463/root (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_6 463 root txt unknown /proc/463/exe (readlink: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_6 463 root NOFD /proc/463/fd (opendir: Permission denied) 2024-04-22T06:40:17.591 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 464 root cwd unknown /proc/464/cwd (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 464 root rtd unknown /proc/464/root (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 464 root txt unknown /proc/464/exe (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 464 root NOFD /proc/464/fd (opendir: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_7 465 root cwd unknown /proc/465/cwd (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_7 465 root rtd unknown /proc/465/root (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_7 465 root txt unknown /proc/465/exe (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_7 465 root NOFD /proc/465/fd (opendir: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 466 root cwd unknown /proc/466/cwd (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 466 root rtd unknown /proc/466/root (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 466 root txt unknown /proc/466/exe (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 466 root NOFD /proc/466/fd (opendir: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_8 467 root cwd unknown /proc/467/cwd (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_8 467 root rtd unknown /proc/467/root (readlink: Permission denied) 2024-04-22T06:40:17.592 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_8 467 root txt unknown /proc/467/exe (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_8 467 root NOFD /proc/467/fd (opendir: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 468 root cwd unknown /proc/468/cwd (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 468 root rtd unknown /proc/468/root (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 468 root txt unknown /proc/468/exe (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 468 root NOFD /proc/468/fd (opendir: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_9 469 root cwd unknown /proc/469/cwd (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_9 469 root rtd unknown /proc/469/root (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_9 469 root txt unknown /proc/469/exe (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_eh_9 469 root NOFD /proc/469/fd (opendir: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 470 root cwd unknown /proc/470/cwd (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 470 root rtd unknown /proc/470/root (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 470 root txt unknown /proc/470/exe (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:scsi_tmf_ 470 root NOFD /proc/470/fd (opendir: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 471 root cwd unknown /proc/471/cwd (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 471 root rtd unknown /proc/471/root (readlink: Permission denied) 2024-04-22T06:40:17.593 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 471 root txt unknown /proc/471/exe (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 471 root NOFD /proc/471/fd (opendir: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 498 root cwd unknown /proc/498/cwd (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 498 root rtd unknown /proc/498/root (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 498 root txt unknown /proc/498/exe (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 498 root NOFD /proc/498/fd (opendir: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:jbd2/sda1 499 root cwd unknown /proc/499/cwd (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:jbd2/sda1 499 root rtd unknown /proc/499/root (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:jbd2/sda1 499 root txt unknown /proc/499/exe (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:jbd2/sda1 499 root NOFD /proc/499/fd (opendir: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:ext4-rsv- 500 root cwd unknown /proc/500/cwd (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:ext4-rsv- 500 root rtd unknown /proc/500/root (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:ext4-rsv- 500 root txt unknown /proc/500/exe (readlink: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:ext4-rsv- 500 root NOFD /proc/500/fd (opendir: Permission denied) 2024-04-22T06:40:17.594 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-j 575 root cwd unknown /proc/575/cwd (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-j 575 root rtd unknown /proc/575/root (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-j 575 root txt unknown /proc/575/exe (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-j 575 root NOFD /proc/575/fd (opendir: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ext4lazyi 581 root cwd unknown /proc/581/cwd (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ext4lazyi 581 root rtd unknown /proc/581/root (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ext4lazyi 581 root txt unknown /proc/581/exe (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ext4lazyi 581 root NOFD /proc/581/fd (opendir: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-u 594 root cwd unknown /proc/594/cwd (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-u 594 root rtd unknown /proc/594/root (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-u 594 root txt unknown /proc/594/exe (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-u 594 root NOFD /proc/594/fd (opendir: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ipmi-msgh 646 root cwd unknown /proc/646/cwd (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ipmi-msgh 646 root rtd unknown /proc/646/root (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ipmi-msgh 646 root txt unknown /proc/646/exe (readlink: Permission denied) 2024-04-22T06:40:17.595 INFO:tasks.workunit.client.0.smithi105.stdout:ipmi-msgh 646 root NOFD /proc/646/fd (opendir: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:kipmi0 656 root cwd unknown /proc/656/cwd (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:kipmi0 656 root rtd unknown /proc/656/root (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:kipmi0 656 root txt unknown /proc/656/exe (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:kipmi0 656 root NOFD /proc/656/fd (opendir: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:rpcbind 663 rpc cwd unknown /proc/663/cwd (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:rpcbind 663 rpc rtd unknown /proc/663/root (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:rpcbind 663 rpc txt unknown /proc/663/exe (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:rpcbind 663 rpc NOFD /proc/663/fd (opendir: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 root cwd unknown /proc/673/cwd (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 root rtd unknown /proc/673/root (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 root txt unknown /proc/673/exe (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 root NOFD /proc/673/fd (opendir: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 674 auditd root cwd unknown /proc/673/task/674/cwd (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 674 auditd root rtd unknown /proc/673/task/674/root (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 674 auditd root txt unknown /proc/673/task/674/exe (readlink: Permission denied) 2024-04-22T06:40:17.596 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 674 auditd root NOFD /proc/673/task/674/fd (opendir: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 676 auditd root cwd unknown /proc/673/task/676/cwd (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 676 auditd root rtd unknown /proc/673/task/676/root (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 676 auditd root txt unknown /proc/673/task/676/exe (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:auditd 673 676 auditd root NOFD /proc/673/task/676/fd (opendir: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:sedispatc 675 root cwd unknown /proc/675/cwd (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:sedispatc 675 root rtd unknown /proc/675/root (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:sedispatc 675 root txt unknown /proc/675/exe (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:sedispatc 675 root NOFD /proc/675/fd (opendir: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:rpciod 679 root cwd unknown /proc/679/cwd (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:rpciod 679 root rtd unknown /proc/679/root (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:rpciod 679 root txt unknown /proc/679/exe (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:rpciod 679 root NOFD /proc/679/fd (opendir: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:xprtiod 680 root cwd unknown /proc/680/cwd (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:xprtiod 680 root rtd unknown /proc/680/root (readlink: Permission denied) 2024-04-22T06:40:17.597 INFO:tasks.workunit.client.0.smithi105.stdout:xprtiod 680 root txt unknown /proc/680/exe (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:xprtiod 680 root NOFD /proc/680/fd (opendir: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 717 dbus cwd unknown /proc/717/cwd (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 717 dbus rtd unknown /proc/717/root (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 717 dbus txt unknown /proc/717/exe (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 717 dbus NOFD /proc/717/fd (opendir: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 718 dbus cwd unknown /proc/718/cwd (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 718 dbus rtd unknown /proc/718/root (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 718 dbus txt unknown /proc/718/exe (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:dbus-brok 718 dbus NOFD /proc/718/fd (opendir: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 root cwd unknown /proc/719/cwd (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 root rtd unknown /proc/719/root (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 root txt unknown /proc/719/exe (readlink: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 root NOFD /proc/719/fd (opendir: Permission denied) 2024-04-22T06:40:17.598 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 768 gmain root cwd unknown /proc/719/task/768/cwd (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 768 gmain root rtd unknown /proc/719/task/768/root (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 768 gmain root txt unknown /proc/719/task/768/exe (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 768 gmain root NOFD /proc/719/task/768/fd (opendir: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 769 gdbus root cwd unknown /proc/719/task/769/cwd (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 769 gdbus root rtd unknown /proc/719/task/769/root (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 769 gdbus root txt unknown /proc/719/task/769/exe (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:NetworkMa 719 769 gdbus root NOFD /proc/719/task/769/fd (opendir: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 root cwd unknown /proc/722/cwd (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 root rtd unknown /proc/722/root (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 root txt unknown /proc/722/exe (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 root NOFD /proc/722/fd (opendir: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 736 gmain root cwd unknown /proc/722/task/736/cwd (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 736 gmain root rtd unknown /proc/722/task/736/root (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 736 gmain root txt unknown /proc/722/task/736/exe (readlink: Permission denied) 2024-04-22T06:40:17.599 INFO:tasks.workunit.client.0.smithi105.stdout:irqbalanc 722 736 gmain root NOFD /proc/722/task/736/fd (opendir: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:lsmd 723 libstoragemgmt cwd unknown /proc/723/cwd (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:lsmd 723 libstoragemgmt rtd unknown /proc/723/root (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:lsmd 723 libstoragemgmt txt unknown /proc/723/exe (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:lsmd 723 libstoragemgmt NOFD /proc/723/fd (opendir: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:mcelog 724 root cwd unknown /proc/724/cwd (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:mcelog 724 root rtd unknown /proc/724/root (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:mcelog 724 root txt unknown /proc/724/exe (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:mcelog 724 root NOFD /proc/724/fd (opendir: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:smartd 725 root cwd unknown /proc/725/cwd (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:smartd 725 root rtd unknown /proc/725/root (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:smartd 725 root txt unknown /proc/725/exe (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:smartd 725 root NOFD /proc/725/fd (opendir: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-l 730 root cwd unknown /proc/730/cwd (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-l 730 root rtd unknown /proc/730/root (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-l 730 root txt unknown /proc/730/exe (readlink: Permission denied) 2024-04-22T06:40:17.600 INFO:tasks.workunit.client.0.smithi105.stdout:systemd-l 730 root NOFD /proc/730/fd (opendir: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 root cwd unknown /proc/818/cwd (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 root rtd unknown /proc/818/root (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 root txt unknown /proc/818/exe (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 root NOFD /proc/818/fd (opendir: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 819 gssproxy root cwd unknown /proc/818/task/819/cwd (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 819 gssproxy root rtd unknown /proc/818/task/819/root (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 819 gssproxy root txt unknown /proc/818/task/819/exe (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 819 gssproxy root NOFD /proc/818/task/819/fd (opendir: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 820 gssproxy root cwd unknown /proc/818/task/820/cwd (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 820 gssproxy root rtd unknown /proc/818/task/820/root (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 820 gssproxy root txt unknown /proc/818/task/820/exe (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 820 gssproxy root NOFD /proc/818/task/820/fd (opendir: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 821 gssproxy root cwd unknown /proc/818/task/821/cwd (readlink: Permission denied) 2024-04-22T06:40:17.601 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 821 gssproxy root rtd unknown /proc/818/task/821/root (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 821 gssproxy root txt unknown /proc/818/task/821/exe (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 821 gssproxy root NOFD /proc/818/task/821/fd (opendir: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 822 gssproxy root cwd unknown /proc/818/task/822/cwd (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 822 gssproxy root rtd unknown /proc/818/task/822/root (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 822 gssproxy root txt unknown /proc/818/task/822/exe (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 822 gssproxy root NOFD /proc/818/task/822/fd (opendir: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 823 gssproxy root cwd unknown /proc/818/task/823/cwd (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 823 gssproxy root rtd unknown /proc/818/task/823/root (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 823 gssproxy root txt unknown /proc/818/task/823/exe (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:gssproxy 818 823 gssproxy root NOFD /proc/818/task/823/fd (opendir: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:atd 825 root cwd unknown /proc/825/cwd (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:atd 825 root rtd unknown /proc/825/root (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:atd 825 root txt unknown /proc/825/exe (readlink: Permission denied) 2024-04-22T06:40:17.602 INFO:tasks.workunit.client.0.smithi105.stdout:atd 825 root NOFD /proc/825/fd (opendir: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:crond 826 root cwd unknown /proc/826/cwd (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:crond 826 root rtd unknown /proc/826/root (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:crond 826 root txt unknown /proc/826/exe (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:crond 826 root NOFD /proc/826/fd (opendir: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 831 root cwd unknown /proc/831/cwd (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 831 root rtd unknown /proc/831/root (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 831 root txt unknown /proc/831/exe (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 831 root NOFD /proc/831/fd (opendir: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:nrpe 853 nrpe cwd unknown /proc/853/cwd (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:nrpe 853 nrpe rtd unknown /proc/853/root (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:nrpe 853 nrpe txt unknown /proc/853/exe (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:nrpe 853 nrpe NOFD /proc/853/fd (opendir: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5189 root cwd unknown /proc/5189/cwd (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5189 root rtd unknown /proc/5189/root (readlink: Permission denied) 2024-04-22T06:40:17.603 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5189 root txt unknown /proc/5189/exe (readlink: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5189 root NOFD /proc/5189/fd (opendir: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5191 root cwd unknown /proc/5191/cwd (readlink: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5191 root rtd unknown /proc/5191/root (readlink: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5191 root txt unknown /proc/5191/exe (readlink: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:agetty 5191 root NOFD /proc/5191/fd (opendir: Permission denied) 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu cwd DIR 8,1 4096 2 / 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu rtd DIR 8,1 4096 2 / 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu txt REG 8,1 98216 18583 /usr/lib/systemd/systemd 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu DEL REG 8,1 61581 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 636848 7222 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 914368 7110 /usr/lib64/libm.so.6 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 882376 7182 /usr/lib64/libzstd.so.1.5.1 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 4487176 7490 /usr/lib64/libcrypto.so.3.0.7 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 1714208 7271 /usr/lib64/libp11-kit.so.0.3.1 2024-04-22T06:40:17.604 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 2592488 7107 /usr/lib64/libc.so.6 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu DEL REG 8,1 62233 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 44784 7268 /usr/lib64/libffi.so.8.1.0 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 153600 7292 /usr/lib64/libgpg-error.so.0.32.0 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 28552 7287 /usr/lib64/libattr.so.1.1.2501 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 102672 7169 /usr/lib64/libz.so.1.2.11 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 32528 7206 /usr/lib64/libcap-ng.so.0.0.0 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 178728 7177 /usr/lib64/liblzma.so.5.2.5 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 1288504 7366 /usr/lib64/libgcrypt.so.20.4.0 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 3785856 46432 /usr/lib64/systemd/libsystemd-shared-252.so 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 40424 8439 /usr/lib64/libeconf.so.0.4.1 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 144120 7350 /usr/lib64/liblz4.so.1.9.3 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 201808 7179 /usr/lib64/libcrypt.so.2.0.0 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 36296 7185 /usr/lib64/libcap.so.2.48 2024-04-22T06:40:17.605 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 2157904 46429 /usr/lib64/systemd/libsystemd-core-252.so 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 229432 7609 /usr/lib64/libblkid.so.1.1.0 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 40496 7288 /usr/lib64/libacl.so.1.1.2301 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 291512 7639 /usr/lib64/libmount.so.1.1.0 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 175752 7307 /usr/lib64/libselinux.so.1 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 107424 7663 /usr/lib64/libkmod.so.2.3.6 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 138848 7210 /usr/lib64/libaudit.so.1.0.0 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 69592 8448 /usr/lib64/libpam.so.0.85.1 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 127680 7455 /usr/lib64/libseccomp.so.2.5.2 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 108136 10561 /usr/lib64/libgcc_s-11-20231218.so.1 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 8,1 867384 11866 /usr/lib64/ld-linux-x86-64.so.2 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 1u unix 0x0000000000000000 0t0 18037 type=STREAM (CONNECTED) 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 2u unix 0x0000000000000000 0t0 18037 type=STREAM (CONNECTED) 2024-04-22T06:40:17.606 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 3u unix 0x0000000000000000 0t0 18065 type=DGRAM (CONNECTED) 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 4u a_inode 0,14 0 2055 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 5u a_inode 0,14 0 2055 [signalfd] 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 6r a_inode 0,14 0 2055 inotify 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 7r DIR 0,25 0 3609 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 8u a_inode 0,14 0 2055 [timerfd] 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 9u a_inode 0,14 0 2055 [eventpoll:10,13] 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 10r REG 0,20 0 18054 /proc/5199/mountinfo 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 11r a_inode 0,14 0 2055 inotify 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 12u unix 0x0000000000000000 0t0 18089 /run/user/1000/bus type=STREAM (LISTEN) 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 13r a_inode 0,14 0 2055 inotify 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 14r REG 0,20 0 4026532076 /proc/swaps 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 15u netlink 0t0 18071 KOBJECT_UEVENT 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 16u unix 0x0000000000000000 0t0 18076 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 17u unix 0x0000000000000000 0t0 18077 type=DGRAM (CONNECTED) 2024-04-22T06:40:17.607 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 18u unix 0x0000000000000000 0t0 18078 type=DGRAM (CONNECTED) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 19u unix 0x0000000000000000 0t0 18079 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 20u unix 0x0000000000000000 0t0 18081 type=STREAM (CONNECTED) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 5199 ubuntu 21u a_inode 0,14 0 2055 [timerfd] 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 5201 ubuntu cwd unknown /proc/5201/cwd (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 5201 ubuntu rtd unknown /proc/5201/root (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 5201 ubuntu txt unknown /proc/5201/exe (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 5201 ubuntu NOFD /proc/5201/fd (opendir: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:tls-strp 5553 root cwd unknown /proc/5553/cwd (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:tls-strp 5553 root rtd unknown /proc/5553/root (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:tls-strp 5553 root txt unknown /proc/5553/exe (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:tls-strp 5553 root NOFD /proc/5553/fd (opendir: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 polkitd cwd unknown /proc/5634/cwd (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 polkitd rtd unknown /proc/5634/root (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 polkitd txt unknown /proc/5634/exe (readlink: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 polkitd NOFD /proc/5634/fd (opendir: Permission denied) 2024-04-22T06:40:17.608 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5636 gmain polkitd cwd unknown /proc/5634/task/5636/cwd (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5636 gmain polkitd rtd unknown /proc/5634/task/5636/root (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5636 gmain polkitd txt unknown /proc/5634/task/5636/exe (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5636 gmain polkitd NOFD /proc/5634/task/5636/fd (opendir: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5637 gdbus polkitd cwd unknown /proc/5634/task/5637/cwd (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5637 gdbus polkitd rtd unknown /proc/5634/task/5637/root (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5637 gdbus polkitd txt unknown /proc/5634/task/5637/exe (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5637 gdbus polkitd NOFD /proc/5634/task/5637/fd (opendir: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5638 JS\x20Hel polkitd cwd unknown /proc/5634/task/5638/cwd (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5638 JS\x20Hel polkitd rtd unknown /proc/5634/task/5638/root (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5638 JS\x20Hel polkitd txt unknown /proc/5634/task/5638/exe (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5638 JS\x20Hel polkitd NOFD /proc/5634/task/5638/fd (opendir: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5639 JS\x20Hel polkitd cwd unknown /proc/5634/task/5639/cwd (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5639 JS\x20Hel polkitd rtd unknown /proc/5634/task/5639/root (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5639 JS\x20Hel polkitd txt unknown /proc/5634/task/5639/exe (readlink: Permission denied) 2024-04-22T06:40:17.609 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5639 JS\x20Hel polkitd NOFD /proc/5634/task/5639/fd (opendir: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5640 JS\x20Hel polkitd cwd unknown /proc/5634/task/5640/cwd (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5640 JS\x20Hel polkitd rtd unknown /proc/5634/task/5640/root (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5640 JS\x20Hel polkitd txt unknown /proc/5634/task/5640/exe (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5640 JS\x20Hel polkitd NOFD /proc/5634/task/5640/fd (opendir: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5641 JS\x20Hel polkitd cwd unknown /proc/5634/task/5641/cwd (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5641 JS\x20Hel polkitd rtd unknown /proc/5634/task/5641/root (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5641 JS\x20Hel polkitd txt unknown /proc/5634/task/5641/exe (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5641 JS\x20Hel polkitd NOFD /proc/5634/task/5641/fd (opendir: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5642 JS\x20Hel polkitd cwd unknown /proc/5634/task/5642/cwd (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5642 JS\x20Hel polkitd rtd unknown /proc/5634/task/5642/root (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5642 JS\x20Hel polkitd txt unknown /proc/5634/task/5642/exe (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5642 JS\x20Hel polkitd NOFD /proc/5634/task/5642/fd (opendir: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5643 JS\x20Hel polkitd cwd unknown /proc/5634/task/5643/cwd (readlink: Permission denied) 2024-04-22T06:40:17.610 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5643 JS\x20Hel polkitd rtd unknown /proc/5634/task/5643/root (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5643 JS\x20Hel polkitd txt unknown /proc/5634/task/5643/exe (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5643 JS\x20Hel polkitd NOFD /proc/5634/task/5643/fd (opendir: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5644 JS\x20Hel polkitd cwd unknown /proc/5634/task/5644/cwd (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5644 JS\x20Hel polkitd rtd unknown /proc/5634/task/5644/root (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5644 JS\x20Hel polkitd txt unknown /proc/5634/task/5644/exe (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5644 JS\x20Hel polkitd NOFD /proc/5634/task/5644/fd (opendir: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5645 JS\x20Hel polkitd cwd unknown /proc/5634/task/5645/cwd (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5645 JS\x20Hel polkitd rtd unknown /proc/5634/task/5645/root (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5645 JS\x20Hel polkitd txt unknown /proc/5634/task/5645/exe (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5645 JS\x20Hel polkitd NOFD /proc/5634/task/5645/fd (opendir: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5646 polkitd polkitd cwd unknown /proc/5634/task/5646/cwd (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5646 polkitd polkitd rtd unknown /proc/5634/task/5646/root (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5646 polkitd polkitd txt unknown /proc/5634/task/5646/exe (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:polkitd 5634 5646 polkitd polkitd NOFD /proc/5634/task/5646/fd (opendir: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 root cwd unknown /proc/5864/cwd (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 root rtd unknown /proc/5864/root (readlink: Permission denied) 2024-04-22T06:40:17.611 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 root txt unknown /proc/5864/exe (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 root NOFD /proc/5864/fd (opendir: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5866 in:imjour root cwd unknown /proc/5864/task/5866/cwd (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5866 in:imjour root rtd unknown /proc/5864/task/5866/root (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5866 in:imjour root txt unknown /proc/5864/task/5866/exe (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5866 in:imjour root NOFD /proc/5864/task/5866/fd (opendir: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5867 rs:main root cwd unknown /proc/5864/task/5867/cwd (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5867 rs:main root rtd unknown /proc/5864/task/5867/root (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5867 rs:main root txt unknown /proc/5864/task/5867/exe (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:rsyslogd 5864 5867 rs:main root NOFD /proc/5864/task/5867/fd (opendir: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19234 root cwd unknown /proc/19234/cwd (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19234 root rtd unknown /proc/19234/root (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19234 root txt unknown /proc/19234/exe (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19234 root NOFD /proc/19234/fd (opendir: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19263 root cwd unknown /proc/19263/cwd (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19263 root rtd unknown /proc/19263/root (readlink: Permission denied) 2024-04-22T06:40:17.612 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19263 root txt unknown /proc/19263/exe (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19263 root NOFD /proc/19263/fd (opendir: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19292 root cwd unknown /proc/19292/cwd (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19292 root rtd unknown /proc/19292/root (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19292 root txt unknown /proc/19292/exe (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19292 root NOFD /proc/19292/fd (opendir: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19321 root cwd unknown /proc/19321/cwd (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19321 root rtd unknown /proc/19321/root (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19321 root txt unknown /proc/19321/exe (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19321 root NOFD /proc/19321/fd (opendir: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19350 root cwd unknown /proc/19350/cwd (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19350 root rtd unknown /proc/19350/root (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19350 root txt unknown /proc/19350/exe (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:kdmflush/ 19350 root NOFD /proc/19350/fd (opendir: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:xfsalloc 19514 root cwd unknown /proc/19514/cwd (readlink: Permission denied) 2024-04-22T06:40:17.613 INFO:tasks.workunit.client.0.smithi105.stdout:xfsalloc 19514 root rtd unknown /proc/19514/root (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfsalloc 19514 root txt unknown /proc/19514/exe (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfsalloc 19514 root NOFD /proc/19514/fd (opendir: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs_mru_c 19515 root cwd unknown /proc/19515/cwd (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs_mru_c 19515 root rtd unknown /proc/19515/root (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs_mru_c 19515 root txt unknown /proc/19515/exe (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs_mru_c 19515 root NOFD /proc/19515/fd (opendir: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-buf/d 19516 root cwd unknown /proc/19516/cwd (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-buf/d 19516 root rtd unknown /proc/19516/root (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-buf/d 19516 root txt unknown /proc/19516/exe (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-buf/d 19516 root NOFD /proc/19516/fd (opendir: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-conv/ 19517 root cwd unknown /proc/19517/cwd (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-conv/ 19517 root rtd unknown /proc/19517/root (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-conv/ 19517 root txt unknown /proc/19517/exe (readlink: Permission denied) 2024-04-22T06:40:17.614 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-conv/ 19517 root NOFD /proc/19517/fd (opendir: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-recla 19518 root cwd unknown /proc/19518/cwd (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-recla 19518 root rtd unknown /proc/19518/root (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-recla 19518 root txt unknown /proc/19518/exe (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-recla 19518 root NOFD /proc/19518/fd (opendir: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-block 19519 root cwd unknown /proc/19519/cwd (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-block 19519 root rtd unknown /proc/19519/root (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-block 19519 root txt unknown /proc/19519/exe (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-block 19519 root NOFD /proc/19519/fd (opendir: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-inode 19520 root cwd unknown /proc/19520/cwd (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-inode 19520 root rtd unknown /proc/19520/root (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-inode 19520 root txt unknown /proc/19520/exe (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-inode 19520 root NOFD /proc/19520/fd (opendir: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-log/d 19521 root cwd unknown /proc/19521/cwd (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-log/d 19521 root rtd unknown /proc/19521/root (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-log/d 19521 root txt unknown /proc/19521/exe (readlink: Permission denied) 2024-04-22T06:40:17.615 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-log/d 19521 root NOFD /proc/19521/fd (opendir: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-cil/d 19522 root cwd unknown /proc/19522/cwd (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-cil/d 19522 root rtd unknown /proc/19522/root (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-cil/d 19522 root txt unknown /proc/19522/exe (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfs-cil/d 19522 root NOFD /proc/19522/fd (opendir: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfsaild/d 19523 root cwd unknown /proc/19523/cwd (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfsaild/d 19523 root rtd unknown /proc/19523/root (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfsaild/d 19523 root txt unknown /proc/19523/exe (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:xfsaild/d 19523 root NOFD /proc/19523/fd (opendir: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20439 root cwd unknown /proc/20439/cwd (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20439 root rtd unknown /proc/20439/root (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20439 root txt unknown /proc/20439/exe (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20439 root NOFD /proc/20439/fd (opendir: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20442 ubuntu cwd unknown /proc/20442/cwd (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20442 ubuntu rtd unknown /proc/20442/root (readlink: Permission denied) 2024-04-22T06:40:17.616 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20442 ubuntu txt unknown /proc/20442/exe (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 20442 ubuntu NOFD /proc/20442/fd (opendir: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:chronyd 20502 chrony cwd unknown /proc/20502/cwd (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:chronyd 20502 chrony rtd unknown /proc/20502/root (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:chronyd 20502 chrony txt unknown /proc/20502/exe (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:chronyd 20502 chrony NOFD /proc/20502/fd (opendir: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 20765 root cwd unknown /proc/20765/cwd (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 20765 root rtd unknown /proc/20765/root (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 20765 root txt unknown /proc/20765/exe (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 20765 root NOFD /proc/20765/fd (opendir: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-cras 20790 ceph cwd unknown /proc/20790/cwd (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-cras 20790 ceph rtd unknown /proc/20790/root (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-cras 20790 ceph txt unknown /proc/20790/exe (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-cras 20790 ceph NOFD /proc/20790/fd (opendir: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 21422 root cwd unknown /proc/21422/cwd (readlink: Permission denied) 2024-04-22T06:40:17.617 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 21422 root rtd unknown /proc/21422/root (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 21422 root txt unknown /proc/21422/exe (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 21422 root NOFD /proc/21422/fd (opendir: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 26510 root cwd unknown /proc/26510/cwd (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 26510 root rtd unknown /proc/26510/root (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 26510 root txt unknown /proc/26510/exe (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 26510 root NOFD /proc/26510/fd (opendir: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26896 root cwd unknown /proc/26896/cwd (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26896 root rtd unknown /proc/26896/root (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26896 root txt unknown /proc/26896/exe (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26896 root NOFD /proc/26896/fd (opendir: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26913 root cwd unknown /proc/26913/cwd (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26913 root rtd unknown /proc/26913/root (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26913 root txt unknown /proc/26913/exe (readlink: Permission denied) 2024-04-22T06:40:17.618 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26913 root NOFD /proc/26913/fd (opendir: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26914 root cwd unknown /proc/26914/cwd (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26914 root rtd unknown /proc/26914/root (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26914 root txt unknown /proc/26914/exe (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 26914 root NOFD /proc/26914/fd (opendir: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26931 root cwd unknown /proc/26931/cwd (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26931 root rtd unknown /proc/26931/root (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26931 root txt unknown /proc/26931/exe (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 26931 root NOFD /proc/26931/fd (opendir: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 27859 root cwd unknown /proc/27859/cwd (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 27859 root rtd unknown /proc/27859/root (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 27859 root txt unknown /proc/27859/exe (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 27859 root NOFD /proc/27859/fd (opendir: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 27912 root cwd unknown /proc/27912/cwd (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 27912 root rtd unknown /proc/27912/root (readlink: Permission denied) 2024-04-22T06:40:17.619 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 27912 root txt unknown /proc/27912/exe (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 27912 root NOFD /proc/27912/fd (opendir: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 27914 root cwd unknown /proc/27914/cwd (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 27914 root rtd unknown /proc/27914/root (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 27914 root txt unknown /proc/27914/exe (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 27914 root NOFD /proc/27914/fd (opendir: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 ceph cwd unknown /proc/27916/cwd (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 ceph rtd unknown /proc/27916/root (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 ceph txt unknown /proc/27916/exe (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 ceph NOFD /proc/27916/fd (opendir: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27917 log ceph cwd unknown /proc/27916/task/27917/cwd (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27917 log ceph rtd unknown /proc/27916/task/27917/root (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27917 log ceph txt unknown /proc/27916/task/27917/exe (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27917 log ceph NOFD /proc/27916/task/27917/fd (opendir: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27918 service ceph cwd unknown /proc/27916/task/27918/cwd (readlink: Permission denied) 2024-04-22T06:40:17.620 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27918 service ceph rtd unknown /proc/27916/task/27918/root (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27918 service ceph txt unknown /proc/27916/task/27918/exe (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27918 service ceph NOFD /proc/27916/task/27918/fd (opendir: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27919 admin_soc ceph cwd unknown /proc/27916/task/27919/cwd (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27919 admin_soc ceph rtd unknown /proc/27916/task/27919/root (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27919 admin_soc ceph txt unknown /proc/27916/task/27919/exe (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27919 admin_soc ceph NOFD /proc/27916/task/27919/fd (opendir: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27920 signal_ha ceph cwd unknown /proc/27916/task/27920/cwd (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27920 signal_ha ceph rtd unknown /proc/27916/task/27920/root (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27920 signal_ha ceph txt unknown /proc/27916/task/27920/exe (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27920 signal_ha ceph NOFD /proc/27916/task/27920/fd (opendir: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27921 rocksdb:l ceph cwd unknown /proc/27916/task/27921/cwd (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27921 rocksdb:l ceph rtd unknown /proc/27916/task/27921/root (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27921 rocksdb:l ceph txt unknown /proc/27916/task/27921/exe (readlink: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27921 rocksdb:l ceph NOFD /proc/27916/task/27921/fd (opendir: Permission denied) 2024-04-22T06:40:17.621 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27922 rocksdb:h ceph cwd unknown /proc/27916/task/27922/cwd (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27922 rocksdb:h ceph rtd unknown /proc/27916/task/27922/root (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27922 rocksdb:h ceph txt unknown /proc/27916/task/27922/exe (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27922 rocksdb:h ceph NOFD /proc/27916/task/27922/fd (opendir: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27938 ceph-mon ceph cwd unknown /proc/27916/task/27938/cwd (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27938 ceph-mon ceph rtd unknown /proc/27916/task/27938/root (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27938 ceph-mon ceph txt unknown /proc/27916/task/27938/exe (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27938 ceph-mon ceph NOFD /proc/27916/task/27938/fd (opendir: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27939 fn_monsto ceph cwd unknown /proc/27916/task/27939/cwd (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27939 fn_monsto ceph rtd unknown /proc/27916/task/27939/root (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27939 fn_monsto ceph txt unknown /proc/27916/task/27939/exe (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27939 fn_monsto ceph NOFD /proc/27916/task/27939/fd (opendir: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27940 msgr-work ceph cwd unknown /proc/27916/task/27940/cwd (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27940 msgr-work ceph rtd unknown /proc/27916/task/27940/root (readlink: Permission denied) 2024-04-22T06:40:17.622 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27940 msgr-work ceph txt unknown /proc/27916/task/27940/exe (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27940 msgr-work ceph NOFD /proc/27916/task/27940/fd (opendir: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27941 msgr-work ceph cwd unknown /proc/27916/task/27941/cwd (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27941 msgr-work ceph rtd unknown /proc/27916/task/27941/root (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27941 msgr-work ceph txt unknown /proc/27916/task/27941/exe (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27941 msgr-work ceph NOFD /proc/27916/task/27941/fd (opendir: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27942 msgr-work ceph cwd unknown /proc/27916/task/27942/cwd (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27942 msgr-work ceph rtd unknown /proc/27916/task/27942/root (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27942 msgr-work ceph txt unknown /proc/27916/task/27942/exe (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27942 msgr-work ceph NOFD /proc/27916/task/27942/fd (opendir: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27943 OpHistory ceph cwd unknown /proc/27916/task/27943/cwd (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27943 OpHistory ceph rtd unknown /proc/27916/task/27943/root (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27943 OpHistory ceph txt unknown /proc/27916/task/27943/exe (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27943 OpHistory ceph NOFD /proc/27916/task/27943/fd (opendir: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27944 fin ceph cwd unknown /proc/27916/task/27944/cwd (readlink: Permission denied) 2024-04-22T06:40:17.623 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27944 fin ceph rtd unknown /proc/27916/task/27944/root (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27944 fin ceph txt unknown /proc/27916/task/27944/exe (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27944 fin ceph NOFD /proc/27916/task/27944/fd (opendir: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27945 safe_time ceph cwd unknown /proc/27916/task/27945/cwd (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27945 safe_time ceph rtd unknown /proc/27916/task/27945/root (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27945 safe_time ceph txt unknown /proc/27916/task/27945/exe (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27945 safe_time ceph NOFD /proc/27916/task/27945/fd (opendir: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27946 cpu_tp ceph cwd unknown /proc/27916/task/27946/cwd (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27946 cpu_tp ceph rtd unknown /proc/27916/task/27946/root (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27946 cpu_tp ceph txt unknown /proc/27916/task/27946/exe (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27946 cpu_tp ceph NOFD /proc/27916/task/27946/fd (opendir: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27947 cpu_tp ceph cwd unknown /proc/27916/task/27947/cwd (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27947 cpu_tp ceph rtd unknown /proc/27916/task/27947/root (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27947 cpu_tp ceph txt unknown /proc/27916/task/27947/exe (readlink: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27947 cpu_tp ceph NOFD /proc/27916/task/27947/fd (opendir: Permission denied) 2024-04-22T06:40:17.624 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27948 cpu_tp ceph cwd unknown /proc/27916/task/27948/cwd (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27948 cpu_tp ceph rtd unknown /proc/27916/task/27948/root (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27948 cpu_tp ceph txt unknown /proc/27916/task/27948/exe (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27948 cpu_tp ceph NOFD /proc/27916/task/27948/fd (opendir: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27949 cpu_tp ceph cwd unknown /proc/27916/task/27949/cwd (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27949 cpu_tp ceph rtd unknown /proc/27916/task/27949/root (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27949 cpu_tp ceph txt unknown /proc/27916/task/27949/exe (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27949 cpu_tp ceph NOFD /proc/27916/task/27949/fd (opendir: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27950 ms_dispat ceph cwd unknown /proc/27916/task/27950/cwd (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27950 ms_dispat ceph rtd unknown /proc/27916/task/27950/root (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27950 ms_dispat ceph txt unknown /proc/27916/task/27950/exe (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27950 ms_dispat ceph NOFD /proc/27916/task/27950/fd (opendir: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27951 ms_local ceph cwd unknown /proc/27916/task/27951/cwd (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27951 ms_local ceph rtd unknown /proc/27916/task/27951/root (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27951 ms_local ceph txt unknown /proc/27916/task/27951/exe (readlink: Permission denied) 2024-04-22T06:40:17.625 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27951 ms_local ceph NOFD /proc/27916/task/27951/fd (opendir: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27952 safe_time ceph cwd unknown /proc/27916/task/27952/cwd (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27952 safe_time ceph rtd unknown /proc/27916/task/27952/root (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27952 safe_time ceph txt unknown /proc/27916/task/27952/exe (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27952 safe_time ceph NOFD /proc/27916/task/27952/fd (opendir: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27953 ms_dispat ceph cwd unknown /proc/27916/task/27953/cwd (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27953 ms_dispat ceph rtd unknown /proc/27916/task/27953/root (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27953 ms_dispat ceph txt unknown /proc/27916/task/27953/exe (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27953 ms_dispat ceph NOFD /proc/27916/task/27953/fd (opendir: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27954 ms_local ceph cwd unknown /proc/27916/task/27954/cwd (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27954 ms_local ceph rtd unknown /proc/27916/task/27954/root (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27954 ms_local ceph txt unknown /proc/27916/task/27954/exe (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 27954 ms_local ceph NOFD /proc/27916/task/27954/fd (opendir: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 67448 rstore_co ceph cwd unknown /proc/27916/task/67448/cwd (readlink: Permission denied) 2024-04-22T06:40:17.626 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 67448 rstore_co ceph rtd unknown /proc/27916/task/67448/root (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 67448 rstore_co ceph txt unknown /proc/27916/task/67448/exe (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mon 27916 67448 rstore_co ceph NOFD /proc/27916/task/67448/fd (opendir: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 28147 root cwd unknown /proc/28147/cwd (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 28147 root rtd unknown /proc/28147/root (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 28147 root txt unknown /proc/28147/exe (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 28147 root NOFD /proc/28147/fd (opendir: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 28149 root cwd unknown /proc/28149/cwd (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 28149 root rtd unknown /proc/28149/root (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 28149 root txt unknown /proc/28149/exe (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 28149 root NOFD /proc/28149/fd (opendir: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 ceph cwd unknown /proc/28151/cwd (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 ceph rtd unknown /proc/28151/root (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 ceph txt unknown /proc/28151/exe (readlink: Permission denied) 2024-04-22T06:40:17.627 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 ceph NOFD /proc/28151/fd (opendir: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80124 log ceph cwd unknown /proc/28151/task/80124/cwd (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80124 log ceph rtd unknown /proc/28151/task/80124/root (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80124 log ceph txt unknown /proc/28151/task/80124/exe (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80124 log ceph NOFD /proc/28151/task/80124/fd (opendir: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80126 msgr-work ceph cwd unknown /proc/28151/task/80126/cwd (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80126 msgr-work ceph rtd unknown /proc/28151/task/80126/root (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80126 msgr-work ceph txt unknown /proc/28151/task/80126/exe (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80126 msgr-work ceph NOFD /proc/28151/task/80126/fd (opendir: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80127 msgr-work ceph cwd unknown /proc/28151/task/80127/cwd (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80127 msgr-work ceph rtd unknown /proc/28151/task/80127/root (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80127 msgr-work ceph txt unknown /proc/28151/task/80127/exe (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80127 msgr-work ceph NOFD /proc/28151/task/80127/fd (opendir: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80128 msgr-work ceph cwd unknown /proc/28151/task/80128/cwd (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80128 msgr-work ceph rtd unknown /proc/28151/task/80128/root (readlink: Permission denied) 2024-04-22T06:40:17.628 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80128 msgr-work ceph txt unknown /proc/28151/task/80128/exe (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80128 msgr-work ceph NOFD /proc/28151/task/80128/fd (opendir: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80132 service ceph cwd unknown /proc/28151/task/80132/cwd (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80132 service ceph rtd unknown /proc/28151/task/80132/root (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80132 service ceph txt unknown /proc/28151/task/80132/exe (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80132 service ceph NOFD /proc/28151/task/80132/fd (opendir: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80133 admin_soc ceph cwd unknown /proc/28151/task/80133/cwd (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80133 admin_soc ceph rtd unknown /proc/28151/task/80133/root (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80133 admin_soc ceph txt unknown /proc/28151/task/80133/exe (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80133 admin_soc ceph NOFD /proc/28151/task/80133/fd (opendir: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80134 ceph_time ceph cwd unknown /proc/28151/task/80134/cwd (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80134 ceph_time ceph rtd unknown /proc/28151/task/80134/root (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80134 ceph_time ceph txt unknown /proc/28151/task/80134/exe (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80134 ceph_time ceph NOFD /proc/28151/task/80134/fd (opendir: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80135 fn_anonym ceph cwd unknown /proc/28151/task/80135/cwd (readlink: Permission denied) 2024-04-22T06:40:17.629 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80135 fn_anonym ceph rtd unknown /proc/28151/task/80135/root (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80135 fn_anonym ceph txt unknown /proc/28151/task/80135/exe (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80135 fn_anonym ceph NOFD /proc/28151/task/80135/fd (opendir: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80136 signal_ha ceph cwd unknown /proc/28151/task/80136/cwd (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80136 signal_ha ceph rtd unknown /proc/28151/task/80136/root (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80136 signal_ha ceph txt unknown /proc/28151/task/80136/exe (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80136 signal_ha ceph NOFD /proc/28151/task/80136/fd (opendir: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80137 mgrsb-fin ceph cwd unknown /proc/28151/task/80137/cwd (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80137 mgrsb-fin ceph rtd unknown /proc/28151/task/80137/root (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80137 mgrsb-fin ceph txt unknown /proc/28151/task/80137/exe (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80137 mgrsb-fin ceph NOFD /proc/28151/task/80137/fd (opendir: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80138 ms_dispat ceph cwd unknown /proc/28151/task/80138/cwd (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80138 ms_dispat ceph rtd unknown /proc/28151/task/80138/root (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80138 ms_dispat ceph txt unknown /proc/28151/task/80138/exe (readlink: Permission denied) 2024-04-22T06:40:17.630 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80138 ms_dispat ceph NOFD /proc/28151/task/80138/fd (opendir: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80139 ms_local ceph cwd unknown /proc/28151/task/80139/cwd (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80139 ms_local ceph rtd unknown /proc/28151/task/80139/root (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80139 ms_local ceph txt unknown /proc/28151/task/80139/exe (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80139 ms_local ceph NOFD /proc/28151/task/80139/fd (opendir: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80140 io_contex ceph cwd unknown /proc/28151/task/80140/cwd (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80140 io_contex ceph rtd unknown /proc/28151/task/80140/root (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80140 io_contex ceph txt unknown /proc/28151/task/80140/exe (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80140 io_contex ceph NOFD /proc/28151/task/80140/fd (opendir: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80141 io_contex ceph cwd unknown /proc/28151/task/80141/cwd (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80141 io_contex ceph rtd unknown /proc/28151/task/80141/root (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80141 io_contex ceph txt unknown /proc/28151/task/80141/exe (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80141 io_contex ceph NOFD /proc/28151/task/80141/fd (opendir: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80142 safe_time ceph cwd unknown /proc/28151/task/80142/cwd (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80142 safe_time ceph rtd unknown /proc/28151/task/80142/root (readlink: Permission denied) 2024-04-22T06:40:17.631 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80142 safe_time ceph txt unknown /proc/28151/task/80142/exe (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80142 safe_time ceph NOFD /proc/28151/task/80142/fd (opendir: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80143 safe_time ceph cwd unknown /proc/28151/task/80143/cwd (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80143 safe_time ceph rtd unknown /proc/28151/task/80143/root (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80143 safe_time ceph txt unknown /proc/28151/task/80143/exe (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80143 safe_time ceph NOFD /proc/28151/task/80143/fd (opendir: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80144 safe_time ceph cwd unknown /proc/28151/task/80144/cwd (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80144 safe_time ceph rtd unknown /proc/28151/task/80144/root (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80144 safe_time ceph txt unknown /proc/28151/task/80144/exe (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80144 safe_time ceph NOFD /proc/28151/task/80144/fd (opendir: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80145 fn_anonym ceph cwd unknown /proc/28151/task/80145/cwd (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80145 fn_anonym ceph rtd unknown /proc/28151/task/80145/root (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80145 fn_anonym ceph txt unknown /proc/28151/task/80145/exe (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80145 fn_anonym ceph NOFD /proc/28151/task/80145/fd (opendir: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80146 flusher ceph cwd unknown /proc/28151/task/80146/cwd (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80146 flusher ceph rtd unknown /proc/28151/task/80146/root (readlink: Permission denied) 2024-04-22T06:40:17.632 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80146 flusher ceph txt unknown /proc/28151/task/80146/exe (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80146 flusher ceph NOFD /proc/28151/task/80146/fd (opendir: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80147 safe_time ceph cwd unknown /proc/28151/task/80147/cwd (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80147 safe_time ceph rtd unknown /proc/28151/task/80147/root (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80147 safe_time ceph txt unknown /proc/28151/task/80147/exe (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80147 safe_time ceph NOFD /proc/28151/task/80147/fd (opendir: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80974 mgr-fin ceph cwd unknown /proc/28151/task/80974/cwd (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80974 mgr-fin ceph rtd unknown /proc/28151/task/80974/root (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80974 mgr-fin ceph txt unknown /proc/28151/task/80974/exe (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80974 mgr-fin ceph NOFD /proc/28151/task/80974/fd (opendir: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80975 ms_dispat ceph cwd unknown /proc/28151/task/80975/cwd (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80975 ms_dispat ceph rtd unknown /proc/28151/task/80975/root (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80975 ms_dispat ceph txt unknown /proc/28151/task/80975/exe (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80975 ms_dispat ceph NOFD /proc/28151/task/80975/fd (opendir: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80976 ms_local ceph cwd unknown /proc/28151/task/80976/cwd (readlink: Permission denied) 2024-04-22T06:40:17.633 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80976 ms_local ceph rtd unknown /proc/28151/task/80976/root (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80976 ms_local ceph txt unknown /proc/28151/task/80976/exe (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80976 ms_local ceph NOFD /proc/28151/task/80976/fd (opendir: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80977 safe_time ceph cwd unknown /proc/28151/task/80977/cwd (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80977 safe_time ceph rtd unknown /proc/28151/task/80977/root (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80977 safe_time ceph txt unknown /proc/28151/task/80977/exe (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80977 safe_time ceph NOFD /proc/28151/task/80977/fd (opendir: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80978 cmdfin ceph cwd unknown /proc/28151/task/80978/cwd (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80978 cmdfin ceph rtd unknown /proc/28151/task/80978/root (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80978 cmdfin ceph txt unknown /proc/28151/task/80978/exe (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80978 cmdfin ceph NOFD /proc/28151/task/80978/fd (opendir: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80982 io_contex ceph cwd unknown /proc/28151/task/80982/cwd (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80982 io_contex ceph rtd unknown /proc/28151/task/80982/root (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80982 io_contex ceph txt unknown /proc/28151/task/80982/exe (readlink: Permission denied) 2024-04-22T06:40:17.634 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80982 io_contex ceph NOFD /proc/28151/task/80982/fd (opendir: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80983 io_contex ceph cwd unknown /proc/28151/task/80983/cwd (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80983 io_contex ceph rtd unknown /proc/28151/task/80983/root (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80983 io_contex ceph txt unknown /proc/28151/task/80983/exe (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80983 io_contex ceph NOFD /proc/28151/task/80983/fd (opendir: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80984 ceph_time ceph cwd unknown /proc/28151/task/80984/cwd (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80984 ceph_time ceph rtd unknown /proc/28151/task/80984/root (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80984 ceph_time ceph txt unknown /proc/28151/task/80984/exe (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80984 ceph_time ceph NOFD /proc/28151/task/80984/fd (opendir: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80985 ms_dispat ceph cwd unknown /proc/28151/task/80985/cwd (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80985 ms_dispat ceph rtd unknown /proc/28151/task/80985/root (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80985 ms_dispat ceph txt unknown /proc/28151/task/80985/exe (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80985 ms_dispat ceph NOFD /proc/28151/task/80985/fd (opendir: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80986 ms_local ceph cwd unknown /proc/28151/task/80986/cwd (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80986 ms_local ceph rtd unknown /proc/28151/task/80986/root (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80986 ms_local ceph txt unknown /proc/28151/task/80986/exe (readlink: Permission denied) 2024-04-22T06:40:17.635 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80986 ms_local ceph NOFD /proc/28151/task/80986/fd (opendir: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80987 safe_time ceph cwd unknown /proc/28151/task/80987/cwd (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80987 safe_time ceph rtd unknown /proc/28151/task/80987/root (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80987 safe_time ceph txt unknown /proc/28151/task/80987/exe (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80987 safe_time ceph NOFD /proc/28151/task/80987/fd (opendir: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80988 safe_time ceph cwd unknown /proc/28151/task/80988/cwd (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80988 safe_time ceph rtd unknown /proc/28151/task/80988/root (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80988 safe_time ceph txt unknown /proc/28151/task/80988/exe (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80988 safe_time ceph NOFD /proc/28151/task/80988/fd (opendir: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80989 balancer ceph cwd unknown /proc/28151/task/80989/cwd (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80989 balancer ceph rtd unknown /proc/28151/task/80989/root (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80989 balancer ceph txt unknown /proc/28151/task/80989/exe (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80989 balancer ceph NOFD /proc/28151/task/80989/fd (opendir: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80990 m-fin-bal ceph cwd unknown /proc/28151/task/80990/cwd (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80990 m-fin-bal ceph rtd unknown /proc/28151/task/80990/root (readlink: Permission denied) 2024-04-22T06:40:17.636 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80990 m-fin-bal ceph txt unknown /proc/28151/task/80990/exe (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80990 m-fin-bal ceph NOFD /proc/28151/task/80990/fd (opendir: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80991 mgr-fin ceph cwd unknown /proc/28151/task/80991/cwd (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80991 mgr-fin ceph rtd unknown /proc/28151/task/80991/root (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80991 mgr-fin ceph txt unknown /proc/28151/task/80991/exe (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80991 mgr-fin ceph NOFD /proc/28151/task/80991/fd (opendir: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80992 mgr-fin ceph cwd unknown /proc/28151/task/80992/cwd (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80992 mgr-fin ceph rtd unknown /proc/28151/task/80992/root (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80992 mgr-fin ceph txt unknown /proc/28151/task/80992/exe (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80992 mgr-fin ceph NOFD /proc/28151/task/80992/fd (opendir: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80993 mgr-fin ceph cwd unknown /proc/28151/task/80993/cwd (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80993 mgr-fin ceph rtd unknown /proc/28151/task/80993/root (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80993 mgr-fin ceph txt unknown /proc/28151/task/80993/exe (readlink: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80993 mgr-fin ceph NOFD /proc/28151/task/80993/fd (opendir: Permission denied) 2024-04-22T06:40:17.637 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80994 mgr-fin ceph cwd unknown /proc/28151/task/80994/cwd (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80994 mgr-fin ceph rtd unknown /proc/28151/task/80994/root (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80994 mgr-fin ceph txt unknown /proc/28151/task/80994/exe (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80994 mgr-fin ceph NOFD /proc/28151/task/80994/fd (opendir: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80995 mgr-fin ceph cwd unknown /proc/28151/task/80995/cwd (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80995 mgr-fin ceph rtd unknown /proc/28151/task/80995/root (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80995 mgr-fin ceph txt unknown /proc/28151/task/80995/exe (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80995 mgr-fin ceph NOFD /proc/28151/task/80995/fd (opendir: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80996 mgr-fin ceph cwd unknown /proc/28151/task/80996/cwd (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80996 mgr-fin ceph rtd unknown /proc/28151/task/80996/root (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80996 mgr-fin ceph txt unknown /proc/28151/task/80996/exe (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80996 mgr-fin ceph NOFD /proc/28151/task/80996/fd (opendir: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80997 mgr-fin ceph cwd unknown /proc/28151/task/80997/cwd (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80997 mgr-fin ceph rtd unknown /proc/28151/task/80997/root (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80997 mgr-fin ceph txt unknown /proc/28151/task/80997/exe (readlink: Permission denied) 2024-04-22T06:40:17.638 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80997 mgr-fin ceph NOFD /proc/28151/task/80997/fd (opendir: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80998 mgr-fin ceph cwd unknown /proc/28151/task/80998/cwd (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80998 mgr-fin ceph rtd unknown /proc/28151/task/80998/root (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80998 mgr-fin ceph txt unknown /proc/28151/task/80998/exe (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80998 mgr-fin ceph NOFD /proc/28151/task/80998/fd (opendir: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80999 mgr-fin ceph cwd unknown /proc/28151/task/80999/cwd (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80999 mgr-fin ceph rtd unknown /proc/28151/task/80999/root (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80999 mgr-fin ceph txt unknown /proc/28151/task/80999/exe (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 80999 mgr-fin ceph NOFD /proc/28151/task/80999/fd (opendir: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81000 mgr-fin ceph cwd unknown /proc/28151/task/81000/cwd (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81000 mgr-fin ceph rtd unknown /proc/28151/task/81000/root (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81000 mgr-fin ceph txt unknown /proc/28151/task/81000/exe (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81000 mgr-fin ceph NOFD /proc/28151/task/81000/fd (opendir: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81001 mgr-fin ceph cwd unknown /proc/28151/task/81001/cwd (readlink: Permission denied) 2024-04-22T06:40:17.639 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81001 mgr-fin ceph rtd unknown /proc/28151/task/81001/root (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81001 mgr-fin ceph txt unknown /proc/28151/task/81001/exe (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81001 mgr-fin ceph NOFD /proc/28151/task/81001/fd (opendir: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81002 mgr-fin ceph cwd unknown /proc/28151/task/81002/cwd (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81002 mgr-fin ceph rtd unknown /proc/28151/task/81002/root (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81002 mgr-fin ceph txt unknown /proc/28151/task/81002/exe (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81002 mgr-fin ceph NOFD /proc/28151/task/81002/fd (opendir: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81003 mgr-fin ceph cwd unknown /proc/28151/task/81003/cwd (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81003 mgr-fin ceph rtd unknown /proc/28151/task/81003/root (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81003 mgr-fin ceph txt unknown /proc/28151/task/81003/exe (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81003 mgr-fin ceph NOFD /proc/28151/task/81003/fd (opendir: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81004 mgr-fin ceph cwd unknown /proc/28151/task/81004/cwd (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81004 mgr-fin ceph rtd unknown /proc/28151/task/81004/root (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81004 mgr-fin ceph txt unknown /proc/28151/task/81004/exe (readlink: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81004 mgr-fin ceph NOFD /proc/28151/task/81004/fd (opendir: Permission denied) 2024-04-22T06:40:17.640 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81005 mgr-fin ceph cwd unknown /proc/28151/task/81005/cwd (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81005 mgr-fin ceph rtd unknown /proc/28151/task/81005/root (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81005 mgr-fin ceph txt unknown /proc/28151/task/81005/exe (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81005 mgr-fin ceph NOFD /proc/28151/task/81005/fd (opendir: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81006 mgr-fin ceph cwd unknown /proc/28151/task/81006/cwd (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81006 mgr-fin ceph rtd unknown /proc/28151/task/81006/root (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81006 mgr-fin ceph txt unknown /proc/28151/task/81006/exe (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81006 mgr-fin ceph NOFD /proc/28151/task/81006/fd (opendir: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81007 cephadm ceph cwd unknown /proc/28151/task/81007/cwd (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81007 cephadm ceph rtd unknown /proc/28151/task/81007/root (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81007 cephadm ceph txt unknown /proc/28151/task/81007/exe (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81007 cephadm ceph NOFD /proc/28151/task/81007/fd (opendir: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81008 m-fin-cep ceph cwd unknown /proc/28151/task/81008/cwd (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81008 m-fin-cep ceph rtd unknown /proc/28151/task/81008/root (readlink: Permission denied) 2024-04-22T06:40:17.641 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81008 m-fin-cep ceph txt unknown /proc/28151/task/81008/exe (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81008 m-fin-cep ceph NOFD /proc/28151/task/81008/fd (opendir: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81009 mgr-fin ceph cwd unknown /proc/28151/task/81009/cwd (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81009 mgr-fin ceph rtd unknown /proc/28151/task/81009/root (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81009 mgr-fin ceph txt unknown /proc/28151/task/81009/exe (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81009 mgr-fin ceph NOFD /proc/28151/task/81009/fd (opendir: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81010 crash ceph cwd unknown /proc/28151/task/81010/cwd (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81010 crash ceph rtd unknown /proc/28151/task/81010/root (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81010 crash ceph txt unknown /proc/28151/task/81010/exe (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81010 crash ceph NOFD /proc/28151/task/81010/fd (opendir: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81011 m-fin-cra ceph cwd unknown /proc/28151/task/81011/cwd (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81011 m-fin-cra ceph rtd unknown /proc/28151/task/81011/root (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81011 m-fin-cra ceph txt unknown /proc/28151/task/81011/exe (readlink: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81011 m-fin-cra ceph NOFD /proc/28151/task/81011/fd (opendir: Permission denied) 2024-04-22T06:40:17.642 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81012 mgr-fin ceph cwd unknown /proc/28151/task/81012/cwd (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81012 mgr-fin ceph rtd unknown /proc/28151/task/81012/root (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81012 mgr-fin ceph txt unknown /proc/28151/task/81012/exe (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81012 mgr-fin ceph NOFD /proc/28151/task/81012/fd (opendir: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81013 cephadm ceph cwd unknown /proc/28151/task/81013/cwd (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81013 cephadm ceph rtd unknown /proc/28151/task/81013/root (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81013 cephadm ceph txt unknown /proc/28151/task/81013/exe (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81013 cephadm ceph NOFD /proc/28151/task/81013/fd (opendir: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81014 mgr-fin ceph cwd unknown /proc/28151/task/81014/cwd (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81014 mgr-fin ceph rtd unknown /proc/28151/task/81014/root (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81014 mgr-fin ceph txt unknown /proc/28151/task/81014/exe (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81014 mgr-fin ceph NOFD /proc/28151/task/81014/fd (opendir: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81015 dashboard ceph cwd unknown /proc/28151/task/81015/cwd (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81015 dashboard ceph rtd unknown /proc/28151/task/81015/root (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81015 dashboard ceph txt unknown /proc/28151/task/81015/exe (readlink: Permission denied) 2024-04-22T06:40:17.643 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81015 dashboard ceph NOFD /proc/28151/task/81015/fd (opendir: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81016 m-fin-das ceph cwd unknown /proc/28151/task/81016/cwd (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81016 m-fin-das ceph rtd unknown /proc/28151/task/81016/root (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81016 m-fin-das ceph txt unknown /proc/28151/task/81016/exe (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81016 m-fin-das ceph NOFD /proc/28151/task/81016/fd (opendir: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81017 mgr-fin ceph cwd unknown /proc/28151/task/81017/cwd (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81017 mgr-fin ceph rtd unknown /proc/28151/task/81017/root (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81017 mgr-fin ceph txt unknown /proc/28151/task/81017/exe (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81017 mgr-fin ceph NOFD /proc/28151/task/81017/fd (opendir: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81018 devicehea ceph cwd unknown /proc/28151/task/81018/cwd (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81018 devicehea ceph rtd unknown /proc/28151/task/81018/root (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81018 devicehea ceph txt unknown /proc/28151/task/81018/exe (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81018 devicehea ceph NOFD /proc/28151/task/81018/fd (opendir: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81019 m-fin-dev ceph cwd unknown /proc/28151/task/81019/cwd (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81019 m-fin-dev ceph rtd unknown /proc/28151/task/81019/root (readlink: Permission denied) 2024-04-22T06:40:17.644 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81019 m-fin-dev ceph txt unknown /proc/28151/task/81019/exe (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81019 m-fin-dev ceph NOFD /proc/28151/task/81019/fd (opendir: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81020 mgr-fin ceph cwd unknown /proc/28151/task/81020/cwd (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81020 mgr-fin ceph rtd unknown /proc/28151/task/81020/root (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81020 mgr-fin ceph txt unknown /proc/28151/task/81020/exe (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81020 mgr-fin ceph NOFD /proc/28151/task/81020/fd (opendir: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81022 m-fin-ios ceph cwd unknown /proc/28151/task/81022/cwd (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81022 m-fin-ios ceph rtd unknown /proc/28151/task/81022/root (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81022 m-fin-ios ceph txt unknown /proc/28151/task/81022/exe (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81022 m-fin-ios ceph NOFD /proc/28151/task/81022/fd (opendir: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81023 mgr-fin ceph cwd unknown /proc/28151/task/81023/cwd (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81023 mgr-fin ceph rtd unknown /proc/28151/task/81023/root (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81023 mgr-fin ceph txt unknown /proc/28151/task/81023/exe (readlink: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81023 mgr-fin ceph NOFD /proc/28151/task/81023/fd (opendir: Permission denied) 2024-04-22T06:40:17.645 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81024 mgr-fin ceph cwd unknown /proc/28151/task/81024/cwd (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81024 mgr-fin ceph rtd unknown /proc/28151/task/81024/root (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81024 mgr-fin ceph txt unknown /proc/28151/task/81024/exe (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81024 mgr-fin ceph NOFD /proc/28151/task/81024/fd (opendir: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81026 m-fin-nfs ceph cwd unknown /proc/28151/task/81026/cwd (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81026 m-fin-nfs ceph rtd unknown /proc/28151/task/81026/root (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81026 m-fin-nfs ceph txt unknown /proc/28151/task/81026/exe (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81026 m-fin-nfs ceph NOFD /proc/28151/task/81026/fd (opendir: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81027 mgr-fin ceph cwd unknown /proc/28151/task/81027/cwd (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81027 mgr-fin ceph rtd unknown /proc/28151/task/81027/root (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81027 mgr-fin ceph txt unknown /proc/28151/task/81027/exe (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81027 mgr-fin ceph NOFD /proc/28151/task/81027/fd (opendir: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81028 mgr-fin ceph cwd unknown /proc/28151/task/81028/cwd (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81028 mgr-fin ceph rtd unknown /proc/28151/task/81028/root (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81028 mgr-fin ceph txt unknown /proc/28151/task/81028/exe (readlink: Permission denied) 2024-04-22T06:40:17.646 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81028 mgr-fin ceph NOFD /proc/28151/task/81028/fd (opendir: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81029 mgr-fin ceph cwd unknown /proc/28151/task/81029/cwd (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81029 mgr-fin ceph rtd unknown /proc/28151/task/81029/root (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81029 mgr-fin ceph txt unknown /proc/28151/task/81029/exe (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81029 mgr-fin ceph NOFD /proc/28151/task/81029/fd (opendir: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81031 m-fin-orc ceph cwd unknown /proc/28151/task/81031/cwd (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81031 m-fin-orc ceph rtd unknown /proc/28151/task/81031/root (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81031 m-fin-orc ceph txt unknown /proc/28151/task/81031/exe (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81031 m-fin-orc ceph NOFD /proc/28151/task/81031/fd (opendir: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81032 mgr-fin ceph cwd unknown /proc/28151/task/81032/cwd (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81032 mgr-fin ceph rtd unknown /proc/28151/task/81032/root (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81032 mgr-fin ceph txt unknown /proc/28151/task/81032/exe (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81032 mgr-fin ceph NOFD /proc/28151/task/81032/fd (opendir: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81033 mgr-fin ceph cwd unknown /proc/28151/task/81033/cwd (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81033 mgr-fin ceph rtd unknown /proc/28151/task/81033/root (readlink: Permission denied) 2024-04-22T06:40:17.647 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81033 mgr-fin ceph txt unknown /proc/28151/task/81033/exe (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81033 mgr-fin ceph NOFD /proc/28151/task/81033/fd (opendir: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81034 pg_autosc ceph cwd unknown /proc/28151/task/81034/cwd (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81034 pg_autosc ceph rtd unknown /proc/28151/task/81034/root (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81034 pg_autosc ceph txt unknown /proc/28151/task/81034/exe (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81034 pg_autosc ceph NOFD /proc/28151/task/81034/fd (opendir: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81035 m-fin-pg_ ceph cwd unknown /proc/28151/task/81035/cwd (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81035 m-fin-pg_ ceph rtd unknown /proc/28151/task/81035/root (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81035 m-fin-pg_ ceph txt unknown /proc/28151/task/81035/exe (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81035 m-fin-pg_ ceph NOFD /proc/28151/task/81035/fd (opendir: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81036 progress ceph cwd unknown /proc/28151/task/81036/cwd (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81036 progress ceph rtd unknown /proc/28151/task/81036/root (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81036 progress ceph txt unknown /proc/28151/task/81036/exe (readlink: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81036 progress ceph NOFD /proc/28151/task/81036/fd (opendir: Permission denied) 2024-04-22T06:40:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81037 m-fin-pro ceph cwd unknown /proc/28151/task/81037/cwd (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81037 m-fin-pro ceph rtd unknown /proc/28151/task/81037/root (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81037 m-fin-pro ceph txt unknown /proc/28151/task/81037/exe (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81037 m-fin-pro ceph NOFD /proc/28151/task/81037/fd (opendir: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81041 io_contex ceph cwd unknown /proc/28151/task/81041/cwd (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81041 io_contex ceph rtd unknown /proc/28151/task/81041/root (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81041 io_contex ceph txt unknown /proc/28151/task/81041/exe (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81041 io_contex ceph NOFD /proc/28151/task/81041/fd (opendir: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81042 io_contex ceph cwd unknown /proc/28151/task/81042/cwd (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81042 io_contex ceph rtd unknown /proc/28151/task/81042/root (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81042 io_contex ceph txt unknown /proc/28151/task/81042/exe (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81042 io_contex ceph NOFD /proc/28151/task/81042/fd (opendir: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81043 ceph_time ceph cwd unknown /proc/28151/task/81043/cwd (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81043 ceph_time ceph rtd unknown /proc/28151/task/81043/root (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81043 ceph_time ceph txt unknown /proc/28151/task/81043/exe (readlink: Permission denied) 2024-04-22T06:40:17.649 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81043 ceph_time ceph NOFD /proc/28151/task/81043/fd (opendir: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81044 ms_dispat ceph cwd unknown /proc/28151/task/81044/cwd (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81044 ms_dispat ceph rtd unknown /proc/28151/task/81044/root (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81044 ms_dispat ceph txt unknown /proc/28151/task/81044/exe (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81044 ms_dispat ceph NOFD /proc/28151/task/81044/fd (opendir: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81045 ms_local ceph cwd unknown /proc/28151/task/81045/cwd (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81045 ms_local ceph rtd unknown /proc/28151/task/81045/root (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81045 ms_local ceph txt unknown /proc/28151/task/81045/exe (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81045 ms_local ceph NOFD /proc/28151/task/81045/fd (opendir: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81046 safe_time ceph cwd unknown /proc/28151/task/81046/cwd (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81046 safe_time ceph rtd unknown /proc/28151/task/81046/root (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81046 safe_time ceph txt unknown /proc/28151/task/81046/exe (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81046 safe_time ceph NOFD /proc/28151/task/81046/fd (opendir: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81047 safe_time ceph cwd unknown /proc/28151/task/81047/cwd (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81047 safe_time ceph rtd unknown /proc/28151/task/81047/root (readlink: Permission denied) 2024-04-22T06:40:17.650 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81047 safe_time ceph txt unknown /proc/28151/task/81047/exe (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81047 safe_time ceph NOFD /proc/28151/task/81047/fd (opendir: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81048 mgr-fin ceph cwd unknown /proc/28151/task/81048/cwd (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81048 mgr-fin ceph rtd unknown /proc/28151/task/81048/root (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81048 mgr-fin ceph txt unknown /proc/28151/task/81048/exe (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81048 mgr-fin ceph NOFD /proc/28151/task/81048/fd (opendir: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81050 m-fin-rbd ceph cwd unknown /proc/28151/task/81050/cwd (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81050 m-fin-rbd ceph rtd unknown /proc/28151/task/81050/root (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81050 m-fin-rbd ceph txt unknown /proc/28151/task/81050/exe (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81050 m-fin-rbd ceph NOFD /proc/28151/task/81050/fd (opendir: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81051 restful ceph cwd unknown /proc/28151/task/81051/cwd (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81051 restful ceph rtd unknown /proc/28151/task/81051/root (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81051 restful ceph txt unknown /proc/28151/task/81051/exe (readlink: Permission denied) 2024-04-22T06:40:17.651 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81051 restful ceph NOFD /proc/28151/task/81051/fd (opendir: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81052 m-fin-res ceph cwd unknown /proc/28151/task/81052/cwd (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81052 m-fin-res ceph rtd unknown /proc/28151/task/81052/root (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81052 m-fin-res ceph txt unknown /proc/28151/task/81052/exe (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81052 m-fin-res ceph NOFD /proc/28151/task/81052/fd (opendir: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81053 mgr-fin ceph cwd unknown /proc/28151/task/81053/cwd (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81053 mgr-fin ceph rtd unknown /proc/28151/task/81053/root (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81053 mgr-fin ceph txt unknown /proc/28151/task/81053/exe (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81053 mgr-fin ceph NOFD /proc/28151/task/81053/fd (opendir: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81054 mgr-fin ceph cwd unknown /proc/28151/task/81054/cwd (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81054 mgr-fin ceph rtd unknown /proc/28151/task/81054/root (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81054 mgr-fin ceph txt unknown /proc/28151/task/81054/exe (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81054 mgr-fin ceph NOFD /proc/28151/task/81054/fd (opendir: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81055 mgr-fin ceph cwd unknown /proc/28151/task/81055/cwd (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81055 mgr-fin ceph rtd unknown /proc/28151/task/81055/root (readlink: Permission denied) 2024-04-22T06:40:17.652 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81055 mgr-fin ceph txt unknown /proc/28151/task/81055/exe (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81055 mgr-fin ceph NOFD /proc/28151/task/81055/fd (opendir: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81056 mgr-fin ceph cwd unknown /proc/28151/task/81056/cwd (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81056 mgr-fin ceph rtd unknown /proc/28151/task/81056/root (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81056 mgr-fin ceph txt unknown /proc/28151/task/81056/exe (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81056 mgr-fin ceph NOFD /proc/28151/task/81056/fd (opendir: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81057 mgr-fin ceph cwd unknown /proc/28151/task/81057/cwd (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81057 mgr-fin ceph rtd unknown /proc/28151/task/81057/root (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81057 mgr-fin ceph txt unknown /proc/28151/task/81057/exe (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81057 mgr-fin ceph NOFD /proc/28151/task/81057/fd (opendir: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81058 mgr-fin ceph cwd unknown /proc/28151/task/81058/cwd (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81058 mgr-fin ceph rtd unknown /proc/28151/task/81058/root (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81058 mgr-fin ceph txt unknown /proc/28151/task/81058/exe (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81058 mgr-fin ceph NOFD /proc/28151/task/81058/fd (opendir: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81062 io_contex ceph cwd unknown /proc/28151/task/81062/cwd (readlink: Permission denied) 2024-04-22T06:40:17.653 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81062 io_contex ceph rtd unknown /proc/28151/task/81062/root (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81062 io_contex ceph txt unknown /proc/28151/task/81062/exe (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81062 io_contex ceph NOFD /proc/28151/task/81062/fd (opendir: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81063 io_contex ceph cwd unknown /proc/28151/task/81063/cwd (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81063 io_contex ceph rtd unknown /proc/28151/task/81063/root (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81063 io_contex ceph txt unknown /proc/28151/task/81063/exe (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81063 io_contex ceph NOFD /proc/28151/task/81063/fd (opendir: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81064 ceph_time ceph cwd unknown /proc/28151/task/81064/cwd (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81064 ceph_time ceph rtd unknown /proc/28151/task/81064/root (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81064 ceph_time ceph txt unknown /proc/28151/task/81064/exe (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81064 ceph_time ceph NOFD /proc/28151/task/81064/fd (opendir: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81065 ms_dispat ceph cwd unknown /proc/28151/task/81065/cwd (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81065 ms_dispat ceph rtd unknown /proc/28151/task/81065/root (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81065 ms_dispat ceph txt unknown /proc/28151/task/81065/exe (readlink: Permission denied) 2024-04-22T06:40:17.654 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81065 ms_dispat ceph NOFD /proc/28151/task/81065/fd (opendir: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81066 ms_local ceph cwd unknown /proc/28151/task/81066/cwd (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81066 ms_local ceph rtd unknown /proc/28151/task/81066/root (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81066 ms_local ceph txt unknown /proc/28151/task/81066/exe (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81066 ms_local ceph NOFD /proc/28151/task/81066/fd (opendir: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81067 safe_time ceph cwd unknown /proc/28151/task/81067/cwd (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81067 safe_time ceph rtd unknown /proc/28151/task/81067/root (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81067 safe_time ceph txt unknown /proc/28151/task/81067/exe (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81067 safe_time ceph NOFD /proc/28151/task/81067/fd (opendir: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81068 safe_time ceph cwd unknown /proc/28151/task/81068/cwd (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81068 safe_time ceph rtd unknown /proc/28151/task/81068/root (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81068 safe_time ceph txt unknown /proc/28151/task/81068/exe (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81068 safe_time ceph NOFD /proc/28151/task/81068/fd (opendir: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81072 m-fin-sna ceph cwd unknown /proc/28151/task/81072/cwd (readlink: Permission denied) 2024-04-22T06:40:17.655 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81072 m-fin-sna ceph rtd unknown /proc/28151/task/81072/root (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81072 m-fin-sna ceph txt unknown /proc/28151/task/81072/exe (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81072 m-fin-sna ceph NOFD /proc/28151/task/81072/fd (opendir: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81074 m-fin-sta ceph cwd unknown /proc/28151/task/81074/cwd (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81074 m-fin-sta ceph rtd unknown /proc/28151/task/81074/root (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81074 m-fin-sta ceph txt unknown /proc/28151/task/81074/exe (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81074 m-fin-sta ceph NOFD /proc/28151/task/81074/fd (opendir: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81075 telemetry ceph cwd unknown /proc/28151/task/81075/cwd (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81075 telemetry ceph rtd unknown /proc/28151/task/81075/root (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81075 telemetry ceph txt unknown /proc/28151/task/81075/exe (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81075 telemetry ceph NOFD /proc/28151/task/81075/fd (opendir: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81076 m-fin-tel ceph cwd unknown /proc/28151/task/81076/cwd (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81076 m-fin-tel ceph rtd unknown /proc/28151/task/81076/root (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81076 m-fin-tel ceph txt unknown /proc/28151/task/81076/exe (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81076 m-fin-tel ceph NOFD /proc/28151/task/81076/fd (opendir: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81077 mgr-fin ceph cwd unknown /proc/28151/task/81077/cwd (readlink: Permission denied) 2024-04-22T06:40:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81077 mgr-fin ceph rtd unknown /proc/28151/task/81077/root (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81077 mgr-fin ceph txt unknown /proc/28151/task/81077/exe (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81077 mgr-fin ceph NOFD /proc/28151/task/81077/fd (opendir: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81081 io_contex ceph cwd unknown /proc/28151/task/81081/cwd (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81081 io_contex ceph rtd unknown /proc/28151/task/81081/root (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81081 io_contex ceph txt unknown /proc/28151/task/81081/exe (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81081 io_contex ceph NOFD /proc/28151/task/81081/fd (opendir: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81082 io_contex ceph cwd unknown /proc/28151/task/81082/cwd (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81082 io_contex ceph rtd unknown /proc/28151/task/81082/root (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81082 io_contex ceph txt unknown /proc/28151/task/81082/exe (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81082 io_contex ceph NOFD /proc/28151/task/81082/fd (opendir: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81083 ceph_time ceph cwd unknown /proc/28151/task/81083/cwd (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81083 ceph_time ceph rtd unknown /proc/28151/task/81083/root (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81083 ceph_time ceph txt unknown /proc/28151/task/81083/exe (readlink: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81083 ceph_time ceph NOFD /proc/28151/task/81083/fd (opendir: Permission denied) 2024-04-22T06:40:17.657 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81084 ms_dispat ceph cwd unknown /proc/28151/task/81084/cwd (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81084 ms_dispat ceph rtd unknown /proc/28151/task/81084/root (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81084 ms_dispat ceph txt unknown /proc/28151/task/81084/exe (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81084 ms_dispat ceph NOFD /proc/28151/task/81084/fd (opendir: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81085 ms_local ceph cwd unknown /proc/28151/task/81085/cwd (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81085 ms_local ceph rtd unknown /proc/28151/task/81085/root (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81085 ms_local ceph txt unknown /proc/28151/task/81085/exe (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81085 ms_local ceph NOFD /proc/28151/task/81085/fd (opendir: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81086 safe_time ceph cwd unknown /proc/28151/task/81086/cwd (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81086 safe_time ceph rtd unknown /proc/28151/task/81086/root (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81086 safe_time ceph txt unknown /proc/28151/task/81086/exe (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81086 safe_time ceph NOFD /proc/28151/task/81086/fd (opendir: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81087 safe_time ceph cwd unknown /proc/28151/task/81087/cwd (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81087 safe_time ceph rtd unknown /proc/28151/task/81087/root (readlink: Permission denied) 2024-04-22T06:40:17.658 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81087 safe_time ceph txt unknown /proc/28151/task/81087/exe (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81087 safe_time ceph NOFD /proc/28151/task/81087/fd (opendir: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81088 mgr-fin ceph cwd unknown /proc/28151/task/81088/cwd (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81088 mgr-fin ceph rtd unknown /proc/28151/task/81088/root (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81088 mgr-fin ceph txt unknown /proc/28151/task/81088/exe (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81088 mgr-fin ceph NOFD /proc/28151/task/81088/fd (opendir: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81089 mgr-fin ceph cwd unknown /proc/28151/task/81089/cwd (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81089 mgr-fin ceph rtd unknown /proc/28151/task/81089/root (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81089 mgr-fin ceph txt unknown /proc/28151/task/81089/exe (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81089 mgr-fin ceph NOFD /proc/28151/task/81089/fd (opendir: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81090 mgr-fin ceph cwd unknown /proc/28151/task/81090/cwd (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81090 mgr-fin ceph rtd unknown /proc/28151/task/81090/root (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81090 mgr-fin ceph txt unknown /proc/28151/task/81090/exe (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81090 mgr-fin ceph NOFD /proc/28151/task/81090/fd (opendir: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81091 mgr-fin ceph cwd unknown /proc/28151/task/81091/cwd (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81091 mgr-fin ceph rtd unknown /proc/28151/task/81091/root (readlink: Permission denied) 2024-04-22T06:40:17.659 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81091 mgr-fin ceph txt unknown /proc/28151/task/81091/exe (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81091 mgr-fin ceph NOFD /proc/28151/task/81091/fd (opendir: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81092 mgr-fin ceph cwd unknown /proc/28151/task/81092/cwd (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81092 mgr-fin ceph rtd unknown /proc/28151/task/81092/root (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81092 mgr-fin ceph txt unknown /proc/28151/task/81092/exe (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81092 mgr-fin ceph NOFD /proc/28151/task/81092/fd (opendir: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81093 mgr-fin ceph cwd unknown /proc/28151/task/81093/cwd (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81093 mgr-fin ceph rtd unknown /proc/28151/task/81093/root (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81093 mgr-fin ceph txt unknown /proc/28151/task/81093/exe (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81093 mgr-fin ceph NOFD /proc/28151/task/81093/fd (opendir: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81094 mgr-fin ceph cwd unknown /proc/28151/task/81094/cwd (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81094 mgr-fin ceph rtd unknown /proc/28151/task/81094/root (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81094 mgr-fin ceph txt unknown /proc/28151/task/81094/exe (readlink: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81094 mgr-fin ceph NOFD /proc/28151/task/81094/fd (opendir: Permission denied) 2024-04-22T06:40:17.660 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81095 mgr-fin ceph cwd unknown /proc/28151/task/81095/cwd (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81095 mgr-fin ceph rtd unknown /proc/28151/task/81095/root (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81095 mgr-fin ceph txt unknown /proc/28151/task/81095/exe (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81095 mgr-fin ceph NOFD /proc/28151/task/81095/fd (opendir: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81097 mgr-fin ceph cwd unknown /proc/28151/task/81097/cwd (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81097 mgr-fin ceph rtd unknown /proc/28151/task/81097/root (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81097 mgr-fin ceph txt unknown /proc/28151/task/81097/exe (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81097 mgr-fin ceph NOFD /proc/28151/task/81097/fd (opendir: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81098 mgr-fin ceph cwd unknown /proc/28151/task/81098/cwd (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81098 mgr-fin ceph rtd unknown /proc/28151/task/81098/root (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81098 mgr-fin ceph txt unknown /proc/28151/task/81098/exe (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81098 mgr-fin ceph NOFD /proc/28151/task/81098/fd (opendir: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81099 mgr-fin ceph cwd unknown /proc/28151/task/81099/cwd (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81099 mgr-fin ceph rtd unknown /proc/28151/task/81099/root (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81099 mgr-fin ceph txt unknown /proc/28151/task/81099/exe (readlink: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81099 mgr-fin ceph NOFD /proc/28151/task/81099/fd (opendir: Permission denied) 2024-04-22T06:40:17.661 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81100 mgr-fin ceph cwd unknown /proc/28151/task/81100/cwd (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81100 mgr-fin ceph rtd unknown /proc/28151/task/81100/root (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81100 mgr-fin ceph txt unknown /proc/28151/task/81100/exe (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81100 mgr-fin ceph NOFD /proc/28151/task/81100/fd (opendir: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81102 m-fin-vol ceph cwd unknown /proc/28151/task/81102/cwd (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81102 m-fin-vol ceph rtd unknown /proc/28151/task/81102/root (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81102 m-fin-vol ceph txt unknown /proc/28151/task/81102/exe (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81102 m-fin-vol ceph NOFD /proc/28151/task/81102/fd (opendir: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81103 io_contex ceph cwd unknown /proc/28151/task/81103/cwd (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81103 io_contex ceph rtd unknown /proc/28151/task/81103/root (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81103 io_contex ceph txt unknown /proc/28151/task/81103/exe (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81103 io_contex ceph NOFD /proc/28151/task/81103/fd (opendir: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81104 io_contex ceph cwd unknown /proc/28151/task/81104/cwd (readlink: Permission denied) 2024-04-22T06:40:17.662 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81104 io_contex ceph rtd unknown /proc/28151/task/81104/root (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81104 io_contex ceph txt unknown /proc/28151/task/81104/exe (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81104 io_contex ceph NOFD /proc/28151/task/81104/fd (opendir: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81147 dashboard ceph cwd unknown /proc/28151/task/81147/cwd (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81147 dashboard ceph rtd unknown /proc/28151/task/81147/root (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81147 dashboard ceph txt unknown /proc/28151/task/81147/exe (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81147 dashboard ceph NOFD /proc/28151/task/81147/fd (opendir: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81148 dashboard ceph cwd unknown /proc/28151/task/81148/cwd (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81148 dashboard ceph rtd unknown /proc/28151/task/81148/root (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81148 dashboard ceph txt unknown /proc/28151/task/81148/exe (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81148 dashboard ceph NOFD /proc/28151/task/81148/fd (opendir: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81149 dashboard ceph cwd unknown /proc/28151/task/81149/cwd (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81149 dashboard ceph rtd unknown /proc/28151/task/81149/root (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81149 dashboard ceph txt unknown /proc/28151/task/81149/exe (readlink: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81149 dashboard ceph NOFD /proc/28151/task/81149/fd (opendir: Permission denied) 2024-04-22T06:40:17.663 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81150 dashboard ceph cwd unknown /proc/28151/task/81150/cwd (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81150 dashboard ceph rtd unknown /proc/28151/task/81150/root (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81150 dashboard ceph txt unknown /proc/28151/task/81150/exe (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81150 dashboard ceph NOFD /proc/28151/task/81150/fd (opendir: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81151 dashboard ceph cwd unknown /proc/28151/task/81151/cwd (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81151 dashboard ceph rtd unknown /proc/28151/task/81151/root (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81151 dashboard ceph txt unknown /proc/28151/task/81151/exe (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81151 dashboard ceph NOFD /proc/28151/task/81151/fd (opendir: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81152 dashboard ceph cwd unknown /proc/28151/task/81152/cwd (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81152 dashboard ceph rtd unknown /proc/28151/task/81152/root (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81152 dashboard ceph txt unknown /proc/28151/task/81152/exe (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81152 dashboard ceph NOFD /proc/28151/task/81152/fd (opendir: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81153 dashboard ceph cwd unknown /proc/28151/task/81153/cwd (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81153 dashboard ceph rtd unknown /proc/28151/task/81153/root (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81153 dashboard ceph txt unknown /proc/28151/task/81153/exe (readlink: Permission denied) 2024-04-22T06:40:17.664 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81153 dashboard ceph NOFD /proc/28151/task/81153/fd (opendir: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81154 dashboard ceph cwd unknown /proc/28151/task/81154/cwd (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81154 dashboard ceph rtd unknown /proc/28151/task/81154/root (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81154 dashboard ceph txt unknown /proc/28151/task/81154/exe (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81154 dashboard ceph NOFD /proc/28151/task/81154/fd (opendir: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81155 dashboard ceph cwd unknown /proc/28151/task/81155/cwd (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81155 dashboard ceph rtd unknown /proc/28151/task/81155/root (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81155 dashboard ceph txt unknown /proc/28151/task/81155/exe (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81155 dashboard ceph NOFD /proc/28151/task/81155/fd (opendir: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81156 dashboard ceph cwd unknown /proc/28151/task/81156/cwd (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81156 dashboard ceph rtd unknown /proc/28151/task/81156/root (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81156 dashboard ceph txt unknown /proc/28151/task/81156/exe (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81156 dashboard ceph NOFD /proc/28151/task/81156/fd (opendir: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81157 dashboard ceph cwd unknown /proc/28151/task/81157/cwd (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81157 dashboard ceph rtd unknown /proc/28151/task/81157/root (readlink: Permission denied) 2024-04-22T06:40:17.665 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81157 dashboard ceph txt unknown /proc/28151/task/81157/exe (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81157 dashboard ceph NOFD /proc/28151/task/81157/fd (opendir: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81208 dashboard ceph cwd unknown /proc/28151/task/81208/cwd (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81208 dashboard ceph rtd unknown /proc/28151/task/81208/root (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81208 dashboard ceph txt unknown /proc/28151/task/81208/exe (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81208 dashboard ceph NOFD /proc/28151/task/81208/fd (opendir: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81328 mgr-fin ceph cwd unknown /proc/28151/task/81328/cwd (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81328 mgr-fin ceph rtd unknown /proc/28151/task/81328/root (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81328 mgr-fin ceph txt unknown /proc/28151/task/81328/exe (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81328 mgr-fin ceph NOFD /proc/28151/task/81328/fd (opendir: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81329 mgr-fin ceph cwd unknown /proc/28151/task/81329/cwd (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81329 mgr-fin ceph rtd unknown /proc/28151/task/81329/root (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81329 mgr-fin ceph txt unknown /proc/28151/task/81329/exe (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81329 mgr-fin ceph NOFD /proc/28151/task/81329/fd (opendir: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81330 mgr-fin ceph cwd unknown /proc/28151/task/81330/cwd (readlink: Permission denied) 2024-04-22T06:40:17.666 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81330 mgr-fin ceph rtd unknown /proc/28151/task/81330/root (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81330 mgr-fin ceph txt unknown /proc/28151/task/81330/exe (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81330 mgr-fin ceph NOFD /proc/28151/task/81330/fd (opendir: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81331 mgr-fin ceph cwd unknown /proc/28151/task/81331/cwd (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81331 mgr-fin ceph rtd unknown /proc/28151/task/81331/root (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81331 mgr-fin ceph txt unknown /proc/28151/task/81331/exe (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81331 mgr-fin ceph NOFD /proc/28151/task/81331/fd (opendir: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81332 mgr-fin ceph cwd unknown /proc/28151/task/81332/cwd (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81332 mgr-fin ceph rtd unknown /proc/28151/task/81332/root (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81332 mgr-fin ceph txt unknown /proc/28151/task/81332/exe (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81332 mgr-fin ceph NOFD /proc/28151/task/81332/fd (opendir: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81333 mgr-fin ceph cwd unknown /proc/28151/task/81333/cwd (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81333 mgr-fin ceph rtd unknown /proc/28151/task/81333/root (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81333 mgr-fin ceph txt unknown /proc/28151/task/81333/exe (readlink: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81333 mgr-fin ceph NOFD /proc/28151/task/81333/fd (opendir: Permission denied) 2024-04-22T06:40:17.667 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81334 mgr-fin ceph cwd unknown /proc/28151/task/81334/cwd (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81334 mgr-fin ceph rtd unknown /proc/28151/task/81334/root (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81334 mgr-fin ceph txt unknown /proc/28151/task/81334/exe (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81334 mgr-fin ceph NOFD /proc/28151/task/81334/fd (opendir: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81335 mgr-fin ceph cwd unknown /proc/28151/task/81335/cwd (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81335 mgr-fin ceph rtd unknown /proc/28151/task/81335/root (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81335 mgr-fin ceph txt unknown /proc/28151/task/81335/exe (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81335 mgr-fin ceph NOFD /proc/28151/task/81335/fd (opendir: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81336 mgr-fin ceph cwd unknown /proc/28151/task/81336/cwd (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81336 mgr-fin ceph rtd unknown /proc/28151/task/81336/root (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81336 mgr-fin ceph txt unknown /proc/28151/task/81336/exe (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81336 mgr-fin ceph NOFD /proc/28151/task/81336/fd (opendir: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81337 mgr-fin ceph cwd unknown /proc/28151/task/81337/cwd (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81337 mgr-fin ceph rtd unknown /proc/28151/task/81337/root (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81337 mgr-fin ceph txt unknown /proc/28151/task/81337/exe (readlink: Permission denied) 2024-04-22T06:40:17.668 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81337 mgr-fin ceph NOFD /proc/28151/task/81337/fd (opendir: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81338 mgr-fin ceph cwd unknown /proc/28151/task/81338/cwd (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81338 mgr-fin ceph rtd unknown /proc/28151/task/81338/root (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81338 mgr-fin ceph txt unknown /proc/28151/task/81338/exe (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81338 mgr-fin ceph NOFD /proc/28151/task/81338/fd (opendir: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81365 mgr-fin ceph cwd unknown /proc/28151/task/81365/cwd (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81365 mgr-fin ceph rtd unknown /proc/28151/task/81365/root (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81365 mgr-fin ceph txt unknown /proc/28151/task/81365/exe (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81365 mgr-fin ceph NOFD /proc/28151/task/81365/fd (opendir: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81366 mgr-fin ceph cwd unknown /proc/28151/task/81366/cwd (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81366 mgr-fin ceph rtd unknown /proc/28151/task/81366/root (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81366 mgr-fin ceph txt unknown /proc/28151/task/81366/exe (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81366 mgr-fin ceph NOFD /proc/28151/task/81366/fd (opendir: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81367 mgr-fin ceph cwd unknown /proc/28151/task/81367/cwd (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81367 mgr-fin ceph rtd unknown /proc/28151/task/81367/root (readlink: Permission denied) 2024-04-22T06:40:17.669 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81367 mgr-fin ceph txt unknown /proc/28151/task/81367/exe (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81367 mgr-fin ceph NOFD /proc/28151/task/81367/fd (opendir: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81368 mgr-fin ceph cwd unknown /proc/28151/task/81368/cwd (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81368 mgr-fin ceph rtd unknown /proc/28151/task/81368/root (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81368 mgr-fin ceph txt unknown /proc/28151/task/81368/exe (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81368 mgr-fin ceph NOFD /proc/28151/task/81368/fd (opendir: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81369 mgr-fin ceph cwd unknown /proc/28151/task/81369/cwd (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81369 mgr-fin ceph rtd unknown /proc/28151/task/81369/root (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81369 mgr-fin ceph txt unknown /proc/28151/task/81369/exe (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81369 mgr-fin ceph NOFD /proc/28151/task/81369/fd (opendir: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81370 mgr-fin ceph cwd unknown /proc/28151/task/81370/cwd (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81370 mgr-fin ceph rtd unknown /proc/28151/task/81370/root (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81370 mgr-fin ceph txt unknown /proc/28151/task/81370/exe (readlink: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81370 mgr-fin ceph NOFD /proc/28151/task/81370/fd (opendir: Permission denied) 2024-04-22T06:40:17.670 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81371 mgr-fin ceph cwd unknown /proc/28151/task/81371/cwd (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81371 mgr-fin ceph rtd unknown /proc/28151/task/81371/root (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81371 mgr-fin ceph txt unknown /proc/28151/task/81371/exe (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81371 mgr-fin ceph NOFD /proc/28151/task/81371/fd (opendir: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81372 mgr-fin ceph cwd unknown /proc/28151/task/81372/cwd (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81372 mgr-fin ceph rtd unknown /proc/28151/task/81372/root (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81372 mgr-fin ceph txt unknown /proc/28151/task/81372/exe (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81372 mgr-fin ceph NOFD /proc/28151/task/81372/fd (opendir: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81373 mgr-fin ceph cwd unknown /proc/28151/task/81373/cwd (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81373 mgr-fin ceph rtd unknown /proc/28151/task/81373/root (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81373 mgr-fin ceph txt unknown /proc/28151/task/81373/exe (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81373 mgr-fin ceph NOFD /proc/28151/task/81373/fd (opendir: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81374 mgr-fin ceph cwd unknown /proc/28151/task/81374/cwd (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81374 mgr-fin ceph rtd unknown /proc/28151/task/81374/root (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81374 mgr-fin ceph txt unknown /proc/28151/task/81374/exe (readlink: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81374 mgr-fin ceph NOFD /proc/28151/task/81374/fd (opendir: Permission denied) 2024-04-22T06:40:17.671 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81375 mgr-fin ceph cwd unknown /proc/28151/task/81375/cwd (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81375 mgr-fin ceph rtd unknown /proc/28151/task/81375/root (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81375 mgr-fin ceph txt unknown /proc/28151/task/81375/exe (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81375 mgr-fin ceph NOFD /proc/28151/task/81375/fd (opendir: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81635 devicehea ceph cwd unknown /proc/28151/task/81635/cwd (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81635 devicehea ceph rtd unknown /proc/28151/task/81635/root (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81635 devicehea ceph txt unknown /proc/28151/task/81635/exe (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81635 devicehea ceph NOFD /proc/28151/task/81635/fd (opendir: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81659 io_contex ceph cwd unknown /proc/28151/task/81659/cwd (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81659 io_contex ceph rtd unknown /proc/28151/task/81659/root (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81659 io_contex ceph txt unknown /proc/28151/task/81659/exe (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81659 io_contex ceph NOFD /proc/28151/task/81659/fd (opendir: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81660 io_contex ceph cwd unknown /proc/28151/task/81660/cwd (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81660 io_contex ceph rtd unknown /proc/28151/task/81660/root (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81660 io_contex ceph txt unknown /proc/28151/task/81660/exe (readlink: Permission denied) 2024-04-22T06:40:17.672 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81660 io_contex ceph NOFD /proc/28151/task/81660/fd (opendir: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81661 ceph_time ceph cwd unknown /proc/28151/task/81661/cwd (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81661 ceph_time ceph rtd unknown /proc/28151/task/81661/root (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81661 ceph_time ceph txt unknown /proc/28151/task/81661/exe (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81661 ceph_time ceph NOFD /proc/28151/task/81661/fd (opendir: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81662 ms_dispat ceph cwd unknown /proc/28151/task/81662/cwd (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81662 ms_dispat ceph rtd unknown /proc/28151/task/81662/root (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81662 ms_dispat ceph txt unknown /proc/28151/task/81662/exe (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81662 ms_dispat ceph NOFD /proc/28151/task/81662/fd (opendir: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81663 ms_local ceph cwd unknown /proc/28151/task/81663/cwd (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81663 ms_local ceph rtd unknown /proc/28151/task/81663/root (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81663 ms_local ceph txt unknown /proc/28151/task/81663/exe (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81663 ms_local ceph NOFD /proc/28151/task/81663/fd (opendir: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81664 safe_time ceph cwd unknown /proc/28151/task/81664/cwd (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81664 safe_time ceph rtd unknown /proc/28151/task/81664/root (readlink: Permission denied) 2024-04-22T06:40:17.673 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81664 safe_time ceph txt unknown /proc/28151/task/81664/exe (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81664 safe_time ceph NOFD /proc/28151/task/81664/fd (opendir: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81665 safe_time ceph cwd unknown /proc/28151/task/81665/cwd (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81665 safe_time ceph rtd unknown /proc/28151/task/81665/root (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81665 safe_time ceph txt unknown /proc/28151/task/81665/exe (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 81665 safe_time ceph NOFD /proc/28151/task/81665/fd (opendir: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84891 ceph_time ceph cwd unknown /proc/28151/task/84891/cwd (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84891 ceph_time ceph rtd unknown /proc/28151/task/84891/root (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84891 ceph_time ceph txt unknown /proc/28151/task/84891/exe (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84891 ceph_time ceph NOFD /proc/28151/task/84891/fd (opendir: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84892 fn_anonym ceph cwd unknown /proc/28151/task/84892/cwd (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84892 fn_anonym ceph rtd unknown /proc/28151/task/84892/root (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84892 fn_anonym ceph txt unknown /proc/28151/task/84892/exe (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84892 fn_anonym ceph NOFD /proc/28151/task/84892/fd (opendir: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84893 safe_time ceph cwd unknown /proc/28151/task/84893/cwd (readlink: Permission denied) 2024-04-22T06:40:17.674 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84893 safe_time ceph rtd unknown /proc/28151/task/84893/root (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84893 safe_time ceph txt unknown /proc/28151/task/84893/exe (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84893 safe_time ceph NOFD /proc/28151/task/84893/fd (opendir: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84894 fn_anonym ceph cwd unknown /proc/28151/task/84894/cwd (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84894 fn_anonym ceph rtd unknown /proc/28151/task/84894/root (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84894 fn_anonym ceph txt unknown /proc/28151/task/84894/exe (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84894 fn_anonym ceph NOFD /proc/28151/task/84894/fd (opendir: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84895 flusher ceph cwd unknown /proc/28151/task/84895/cwd (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84895 flusher ceph rtd unknown /proc/28151/task/84895/root (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84895 flusher ceph txt unknown /proc/28151/task/84895/exe (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84895 flusher ceph NOFD /proc/28151/task/84895/fd (opendir: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84896 ms_dispat ceph cwd unknown /proc/28151/task/84896/cwd (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84896 ms_dispat ceph rtd unknown /proc/28151/task/84896/root (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84896 ms_dispat ceph txt unknown /proc/28151/task/84896/exe (readlink: Permission denied) 2024-04-22T06:40:17.675 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84896 ms_dispat ceph NOFD /proc/28151/task/84896/fd (opendir: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84897 ms_local ceph cwd unknown /proc/28151/task/84897/cwd (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84897 ms_local ceph rtd unknown /proc/28151/task/84897/root (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84897 ms_local ceph txt unknown /proc/28151/task/84897/exe (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84897 ms_local ceph NOFD /proc/28151/task/84897/fd (opendir: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84898 safe_time ceph cwd unknown /proc/28151/task/84898/cwd (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84898 safe_time ceph rtd unknown /proc/28151/task/84898/root (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84898 safe_time ceph txt unknown /proc/28151/task/84898/exe (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84898 safe_time ceph NOFD /proc/28151/task/84898/fd (opendir: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84899 m-fin-sna ceph cwd unknown /proc/28151/task/84899/cwd (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84899 m-fin-sna ceph rtd unknown /proc/28151/task/84899/root (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84899 m-fin-sna ceph txt unknown /proc/28151/task/84899/exe (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 84899 m-fin-sna ceph NOFD /proc/28151/task/84899/fd (opendir: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 87948 m-fin-sna ceph cwd unknown /proc/28151/task/87948/cwd (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 87948 m-fin-sna ceph rtd unknown /proc/28151/task/87948/root (readlink: Permission denied) 2024-04-22T06:40:17.676 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 87948 m-fin-sna ceph txt unknown /proc/28151/task/87948/exe (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mgr 28151 87948 m-fin-sna ceph NOFD /proc/28151/task/87948/fd (opendir: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 29120 root cwd unknown /proc/29120/cwd (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 29120 root rtd unknown /proc/29120/root (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 29120 root txt unknown /proc/29120/exe (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 29120 root NOFD /proc/29120/fd (opendir: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 29291 root cwd unknown /proc/29291/cwd (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 29291 root rtd unknown /proc/29291/root (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 29291 root txt unknown /proc/29291/exe (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:systemd 29291 root NOFD /proc/29291/fd (opendir: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 29293 root cwd unknown /proc/29293/cwd (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 29293 root rtd unknown /proc/29293/root (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 29293 root txt unknown /proc/29293/exe (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:(sd-pam) 29293 root NOFD /proc/29293/fd (opendir: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 38055 root cwd unknown /proc/38055/cwd (readlink: Permission denied) 2024-04-22T06:40:17.677 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 38055 root rtd unknown /proc/38055/root (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 38055 root txt unknown /proc/38055/exe (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 38055 root NOFD /proc/38055/fd (opendir: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 40472 root cwd unknown /proc/40472/cwd (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 40472 root rtd unknown /proc/40472/root (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 40472 root txt unknown /proc/40472/exe (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 40472 root NOFD /proc/40472/fd (opendir: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 40570 root cwd unknown /proc/40570/cwd (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 40570 root rtd unknown /proc/40570/root (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 40570 root txt unknown /proc/40570/exe (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/2 40570 root NOFD /proc/40570/fd (opendir: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 42796 root cwd unknown /proc/42796/cwd (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 42796 root rtd unknown /proc/42796/root (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 42796 root txt unknown /proc/42796/exe (readlink: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 42796 root NOFD /proc/42796/fd (opendir: Permission denied) 2024-04-22T06:40:17.678 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 42798 root cwd unknown /proc/42798/cwd (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 42798 root rtd unknown /proc/42798/root (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 42798 root txt unknown /proc/42798/exe (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 42798 root NOFD /proc/42798/fd (opendir: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 ceph cwd unknown /proc/42800/cwd (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 ceph rtd unknown /proc/42800/root (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 ceph txt unknown /proc/42800/exe (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 ceph NOFD /proc/42800/fd (opendir: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42801 log ceph cwd unknown /proc/42800/task/42801/cwd (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42801 log ceph rtd unknown /proc/42800/task/42801/root (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42801 log ceph txt unknown /proc/42800/task/42801/exe (readlink: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42801 log ceph NOFD /proc/42800/task/42801/fd (opendir: Permission denied) 2024-04-22T06:40:17.679 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42803 msgr-work ceph cwd unknown /proc/42800/task/42803/cwd (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42803 msgr-work ceph rtd unknown /proc/42800/task/42803/root (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42803 msgr-work ceph txt unknown /proc/42800/task/42803/exe (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42803 msgr-work ceph NOFD /proc/42800/task/42803/fd (opendir: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42804 msgr-work ceph cwd unknown /proc/42800/task/42804/cwd (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42804 msgr-work ceph rtd unknown /proc/42800/task/42804/root (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42804 msgr-work ceph txt unknown /proc/42800/task/42804/exe (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42804 msgr-work ceph NOFD /proc/42800/task/42804/fd (opendir: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42805 msgr-work ceph cwd unknown /proc/42800/task/42805/cwd (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42805 msgr-work ceph rtd unknown /proc/42800/task/42805/root (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42805 msgr-work ceph txt unknown /proc/42800/task/42805/exe (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42805 msgr-work ceph NOFD /proc/42800/task/42805/fd (opendir: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42809 service ceph cwd unknown /proc/42800/task/42809/cwd (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42809 service ceph rtd unknown /proc/42800/task/42809/root (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42809 service ceph txt unknown /proc/42800/task/42809/exe (readlink: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42809 service ceph NOFD /proc/42800/task/42809/fd (opendir: Permission denied) 2024-04-22T06:40:17.680 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42810 admin_soc ceph cwd unknown /proc/42800/task/42810/cwd (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42810 admin_soc ceph rtd unknown /proc/42800/task/42810/root (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42810 admin_soc ceph txt unknown /proc/42800/task/42810/exe (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42810 admin_soc ceph NOFD /proc/42800/task/42810/fd (opendir: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42953 signal_ha ceph cwd unknown /proc/42800/task/42953/cwd (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42953 signal_ha ceph rtd unknown /proc/42800/task/42953/root (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42953 signal_ha ceph txt unknown /proc/42800/task/42953/exe (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42953 signal_ha ceph NOFD /proc/42800/task/42953/fd (opendir: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42954 io_contex ceph cwd unknown /proc/42800/task/42954/cwd (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42954 io_contex ceph rtd unknown /proc/42800/task/42954/root (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42954 io_contex ceph txt unknown /proc/42800/task/42954/exe (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42954 io_contex ceph NOFD /proc/42800/task/42954/fd (opendir: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42955 io_contex ceph cwd unknown /proc/42800/task/42955/cwd (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42955 io_contex ceph rtd unknown /proc/42800/task/42955/root (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42955 io_contex ceph txt unknown /proc/42800/task/42955/exe (readlink: Permission denied) 2024-04-22T06:40:17.681 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 42955 io_contex ceph NOFD /proc/42800/task/42955/fd (opendir: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43068 OpHistory ceph cwd unknown /proc/42800/task/43068/cwd (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43068 OpHistory ceph rtd unknown /proc/42800/task/43068/root (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43068 OpHistory ceph txt unknown /proc/42800/task/43068/exe (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43068 OpHistory ceph NOFD /proc/42800/task/43068/fd (opendir: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43069 ceph_time ceph cwd unknown /proc/42800/task/43069/cwd (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43069 ceph_time ceph rtd unknown /proc/42800/task/43069/root (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43069 ceph_time ceph txt unknown /proc/42800/task/43069/exe (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43069 ceph_time ceph NOFD /proc/42800/task/43069/fd (opendir: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43118 ceph-osd ceph cwd unknown /proc/42800/task/43118/cwd (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43118 ceph-osd ceph rtd unknown /proc/42800/task/43118/root (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43118 ceph-osd ceph txt unknown /proc/42800/task/43118/exe (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43118 ceph-osd ceph NOFD /proc/42800/task/43118/fd (opendir: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43123 ceph-osd ceph cwd unknown /proc/42800/task/43123/cwd (readlink: Permission denied) 2024-04-22T06:40:17.682 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43123 ceph-osd ceph rtd unknown /proc/42800/task/43123/root (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43123 ceph-osd ceph txt unknown /proc/42800/task/43123/exe (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43123 ceph-osd ceph NOFD /proc/42800/task/43123/fd (opendir: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43134 ceph-osd ceph cwd unknown /proc/42800/task/43134/cwd (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43134 ceph-osd ceph rtd unknown /proc/42800/task/43134/root (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43134 ceph-osd ceph txt unknown /proc/42800/task/43134/exe (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43134 ceph-osd ceph NOFD /proc/42800/task/43134/fd (opendir: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43151 ceph-osd ceph cwd unknown /proc/42800/task/43151/cwd (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43151 ceph-osd ceph rtd unknown /proc/42800/task/43151/root (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43151 ceph-osd ceph txt unknown /proc/42800/task/43151/exe (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43151 ceph-osd ceph NOFD /proc/42800/task/43151/fd (opendir: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43172 ceph-osd ceph cwd unknown /proc/42800/task/43172/cwd (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43172 ceph-osd ceph rtd unknown /proc/42800/task/43172/root (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43172 ceph-osd ceph txt unknown /proc/42800/task/43172/exe (readlink: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43172 ceph-osd ceph NOFD /proc/42800/task/43172/fd (opendir: Permission denied) 2024-04-22T06:40:17.683 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43212 ceph-osd ceph cwd unknown /proc/42800/task/43212/cwd (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43212 ceph-osd ceph rtd unknown /proc/42800/task/43212/root (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43212 ceph-osd ceph txt unknown /proc/42800/task/43212/exe (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43212 ceph-osd ceph NOFD /proc/42800/task/43212/fd (opendir: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43254 ceph-osd ceph cwd unknown /proc/42800/task/43254/cwd (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43254 ceph-osd ceph rtd unknown /proc/42800/task/43254/root (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43254 ceph-osd ceph txt unknown /proc/42800/task/43254/exe (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43254 ceph-osd ceph NOFD /proc/42800/task/43254/fd (opendir: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43262 ceph-osd ceph cwd unknown /proc/42800/task/43262/cwd (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43262 ceph-osd ceph rtd unknown /proc/42800/task/43262/root (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43262 ceph-osd ceph txt unknown /proc/42800/task/43262/exe (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43262 ceph-osd ceph NOFD /proc/42800/task/43262/fd (opendir: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43264 ceph-osd ceph cwd unknown /proc/42800/task/43264/cwd (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43264 ceph-osd ceph rtd unknown /proc/42800/task/43264/root (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43264 ceph-osd ceph txt unknown /proc/42800/task/43264/exe (readlink: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43264 ceph-osd ceph NOFD /proc/42800/task/43264/fd (opendir: Permission denied) 2024-04-22T06:40:17.684 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43265 safe_time ceph cwd unknown /proc/42800/task/43265/cwd (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43265 safe_time ceph rtd unknown /proc/42800/task/43265/root (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43265 safe_time ceph txt unknown /proc/42800/task/43265/exe (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43265 safe_time ceph NOFD /proc/42800/task/43265/fd (opendir: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43266 safe_time ceph cwd unknown /proc/42800/task/43266/cwd (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43266 safe_time ceph rtd unknown /proc/42800/task/43266/root (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43266 safe_time ceph txt unknown /proc/42800/task/43266/exe (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43266 safe_time ceph NOFD /proc/42800/task/43266/fd (opendir: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43267 safe_time ceph cwd unknown /proc/42800/task/43267/cwd (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43267 safe_time ceph rtd unknown /proc/42800/task/43267/root (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43267 safe_time ceph txt unknown /proc/42800/task/43267/exe (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43267 safe_time ceph NOFD /proc/42800/task/43267/fd (opendir: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43268 safe_time ceph cwd unknown /proc/42800/task/43268/cwd (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43268 safe_time ceph rtd unknown /proc/42800/task/43268/root (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43268 safe_time ceph txt unknown /proc/42800/task/43268/exe (readlink: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43268 safe_time ceph NOFD /proc/42800/task/43268/fd (opendir: Permission denied) 2024-04-22T06:40:17.685 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43269 fn_anonym ceph cwd unknown /proc/42800/task/43269/cwd (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43269 fn_anonym ceph rtd unknown /proc/42800/task/43269/root (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43269 fn_anonym ceph txt unknown /proc/42800/task/43269/exe (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43269 fn_anonym ceph NOFD /proc/42800/task/43269/fd (opendir: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43272 rocksdb:l ceph cwd unknown /proc/42800/task/43272/cwd (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43272 rocksdb:l ceph rtd unknown /proc/42800/task/43272/root (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43272 rocksdb:l ceph txt unknown /proc/42800/task/43272/exe (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43272 rocksdb:l ceph NOFD /proc/42800/task/43272/fd (opendir: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43273 rocksdb:l ceph cwd unknown /proc/42800/task/43273/cwd (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43273 rocksdb:l ceph rtd unknown /proc/42800/task/43273/root (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43273 rocksdb:l ceph txt unknown /proc/42800/task/43273/exe (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43273 rocksdb:l ceph NOFD /proc/42800/task/43273/fd (opendir: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43274 rocksdb:l ceph cwd unknown /proc/42800/task/43274/cwd (readlink: Permission denied) 2024-04-22T06:40:17.686 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43274 rocksdb:l ceph rtd unknown /proc/42800/task/43274/root (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43274 rocksdb:l ceph txt unknown /proc/42800/task/43274/exe (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43274 rocksdb:l ceph NOFD /proc/42800/task/43274/fd (opendir: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43275 rocksdb:h ceph cwd unknown /proc/42800/task/43275/cwd (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43275 rocksdb:h ceph rtd unknown /proc/42800/task/43275/root (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43275 rocksdb:h ceph txt unknown /proc/42800/task/43275/exe (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43275 rocksdb:h ceph NOFD /proc/42800/task/43275/fd (opendir: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43681 bstore_ai ceph cwd unknown /proc/42800/task/43681/cwd (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43681 bstore_ai ceph rtd unknown /proc/42800/task/43681/root (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43681 bstore_ai ceph txt unknown /proc/42800/task/43681/exe (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43681 bstore_ai ceph NOFD /proc/42800/task/43681/fd (opendir: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43942 bstore_ai ceph cwd unknown /proc/42800/task/43942/cwd (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43942 bstore_ai ceph rtd unknown /proc/42800/task/43942/root (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43942 bstore_ai ceph txt unknown /proc/42800/task/43942/exe (readlink: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 43942 bstore_ai ceph NOFD /proc/42800/task/43942/fd (opendir: Permission denied) 2024-04-22T06:40:17.687 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44124 ceph-osd ceph cwd unknown /proc/42800/task/44124/cwd (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44124 ceph-osd ceph rtd unknown /proc/42800/task/44124/root (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44124 ceph-osd ceph txt unknown /proc/42800/task/44124/exe (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44124 ceph-osd ceph NOFD /proc/42800/task/44124/fd (opendir: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44125 cfin ceph cwd unknown /proc/42800/task/44125/cwd (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44125 cfin ceph rtd unknown /proc/42800/task/44125/root (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44125 cfin ceph txt unknown /proc/42800/task/44125/exe (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44125 cfin ceph NOFD /proc/42800/task/44125/fd (opendir: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44126 bstore_kv ceph cwd unknown /proc/42800/task/44126/cwd (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44126 bstore_kv ceph rtd unknown /proc/42800/task/44126/root (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44126 bstore_kv ceph txt unknown /proc/42800/task/44126/exe (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44126 bstore_kv ceph NOFD /proc/42800/task/44126/fd (opendir: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44127 bstore_kv ceph cwd unknown /proc/42800/task/44127/cwd (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44127 bstore_kv ceph rtd unknown /proc/42800/task/44127/root (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44127 bstore_kv ceph txt unknown /proc/42800/task/44127/exe (readlink: Permission denied) 2024-04-22T06:40:17.688 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44127 bstore_kv ceph NOFD /proc/42800/task/44127/fd (opendir: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44128 bstore_me ceph cwd unknown /proc/42800/task/44128/cwd (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44128 bstore_me ceph rtd unknown /proc/42800/task/44128/root (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44128 bstore_me ceph txt unknown /proc/42800/task/44128/exe (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44128 bstore_me ceph NOFD /proc/42800/task/44128/fd (opendir: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44129 ms_dispat ceph cwd unknown /proc/42800/task/44129/cwd (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44129 ms_dispat ceph rtd unknown /proc/42800/task/44129/root (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44129 ms_dispat ceph txt unknown /proc/42800/task/44129/exe (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44129 ms_dispat ceph NOFD /proc/42800/task/44129/fd (opendir: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44130 ms_local ceph cwd unknown /proc/42800/task/44130/cwd (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44130 ms_local ceph rtd unknown /proc/42800/task/44130/root (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44130 ms_local ceph txt unknown /proc/42800/task/44130/exe (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44130 ms_local ceph NOFD /proc/42800/task/44130/fd (opendir: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44131 safe_time ceph cwd unknown /proc/42800/task/44131/cwd (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44131 safe_time ceph rtd unknown /proc/42800/task/44131/root (readlink: Permission denied) 2024-04-22T06:40:17.689 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44131 safe_time ceph txt unknown /proc/42800/task/44131/exe (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44131 safe_time ceph NOFD /proc/42800/task/44131/fd (opendir: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44132 safe_time ceph cwd unknown /proc/42800/task/44132/cwd (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44132 safe_time ceph rtd unknown /proc/42800/task/44132/root (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44132 safe_time ceph txt unknown /proc/42800/task/44132/exe (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44132 safe_time ceph NOFD /proc/42800/task/44132/fd (opendir: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44133 ms_dispat ceph cwd unknown /proc/42800/task/44133/cwd (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44133 ms_dispat ceph rtd unknown /proc/42800/task/44133/root (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44133 ms_dispat ceph txt unknown /proc/42800/task/44133/exe (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44133 ms_dispat ceph NOFD /proc/42800/task/44133/fd (opendir: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44134 ms_local ceph cwd unknown /proc/42800/task/44134/cwd (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44134 ms_local ceph rtd unknown /proc/42800/task/44134/root (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44134 ms_local ceph txt unknown /proc/42800/task/44134/exe (readlink: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44134 ms_local ceph NOFD /proc/42800/task/44134/fd (opendir: Permission denied) 2024-04-22T06:40:17.690 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44135 ms_dispat ceph cwd unknown /proc/42800/task/44135/cwd (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44135 ms_dispat ceph rtd unknown /proc/42800/task/44135/root (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44135 ms_dispat ceph txt unknown /proc/42800/task/44135/exe (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44135 ms_dispat ceph NOFD /proc/42800/task/44135/fd (opendir: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44136 ms_local ceph cwd unknown /proc/42800/task/44136/cwd (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44136 ms_local ceph rtd unknown /proc/42800/task/44136/root (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44136 ms_local ceph txt unknown /proc/42800/task/44136/exe (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44136 ms_local ceph NOFD /proc/42800/task/44136/fd (opendir: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44137 ms_dispat ceph cwd unknown /proc/42800/task/44137/cwd (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44137 ms_dispat ceph rtd unknown /proc/42800/task/44137/root (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44137 ms_dispat ceph txt unknown /proc/42800/task/44137/exe (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44137 ms_dispat ceph NOFD /proc/42800/task/44137/fd (opendir: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44138 ms_local ceph cwd unknown /proc/42800/task/44138/cwd (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44138 ms_local ceph rtd unknown /proc/42800/task/44138/root (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44138 ms_local ceph txt unknown /proc/42800/task/44138/exe (readlink: Permission denied) 2024-04-22T06:40:17.691 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44138 ms_local ceph NOFD /proc/42800/task/44138/fd (opendir: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44139 ms_dispat ceph cwd unknown /proc/42800/task/44139/cwd (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44139 ms_dispat ceph rtd unknown /proc/42800/task/44139/root (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44139 ms_dispat ceph txt unknown /proc/42800/task/44139/exe (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44139 ms_dispat ceph NOFD /proc/42800/task/44139/fd (opendir: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44140 ms_local ceph cwd unknown /proc/42800/task/44140/cwd (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44140 ms_local ceph rtd unknown /proc/42800/task/44140/root (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44140 ms_local ceph txt unknown /proc/42800/task/44140/exe (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44140 ms_local ceph NOFD /proc/42800/task/44140/fd (opendir: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44141 ms_dispat ceph cwd unknown /proc/42800/task/44141/cwd (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44141 ms_dispat ceph rtd unknown /proc/42800/task/44141/root (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44141 ms_dispat ceph txt unknown /proc/42800/task/44141/exe (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44141 ms_dispat ceph NOFD /proc/42800/task/44141/fd (opendir: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44142 ms_local ceph cwd unknown /proc/42800/task/44142/cwd (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44142 ms_local ceph rtd unknown /proc/42800/task/44142/root (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44142 ms_local ceph txt unknown /proc/42800/task/44142/exe (readlink: Permission denied) 2024-04-22T06:40:17.692 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44142 ms_local ceph NOFD /proc/42800/task/44142/fd (opendir: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44143 ms_dispat ceph cwd unknown /proc/42800/task/44143/cwd (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44143 ms_dispat ceph rtd unknown /proc/42800/task/44143/root (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44143 ms_dispat ceph txt unknown /proc/42800/task/44143/exe (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44143 ms_dispat ceph NOFD /proc/42800/task/44143/fd (opendir: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44144 ms_local ceph cwd unknown /proc/42800/task/44144/cwd (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44144 ms_local ceph rtd unknown /proc/42800/task/44144/root (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44144 ms_local ceph txt unknown /proc/42800/task/44144/exe (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44144 ms_local ceph NOFD /proc/42800/task/44144/fd (opendir: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44145 fn_anonym ceph cwd unknown /proc/42800/task/44145/cwd (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44145 fn_anonym ceph rtd unknown /proc/42800/task/44145/root (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44145 fn_anonym ceph txt unknown /proc/42800/task/44145/exe (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44145 fn_anonym ceph NOFD /proc/42800/task/44145/fd (opendir: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44146 finisher ceph cwd unknown /proc/42800/task/44146/cwd (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44146 finisher ceph rtd unknown /proc/42800/task/44146/root (readlink: Permission denied) 2024-04-22T06:40:17.693 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44146 finisher ceph txt unknown /proc/42800/task/44146/exe (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44146 finisher ceph NOFD /proc/42800/task/44146/fd (opendir: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44147 safe_time ceph cwd unknown /proc/42800/task/44147/cwd (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44147 safe_time ceph rtd unknown /proc/42800/task/44147/root (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44147 safe_time ceph txt unknown /proc/42800/task/44147/exe (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44147 safe_time ceph NOFD /proc/42800/task/44147/fd (opendir: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44148 safe_time ceph cwd unknown /proc/42800/task/44148/cwd (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44148 safe_time ceph rtd unknown /proc/42800/task/44148/root (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44148 safe_time ceph txt unknown /proc/42800/task/44148/exe (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44148 safe_time ceph NOFD /proc/42800/task/44148/fd (opendir: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44149 ceph-osd ceph cwd unknown /proc/42800/task/44149/cwd (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44149 ceph-osd ceph rtd unknown /proc/42800/task/44149/root (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44149 ceph-osd ceph txt unknown /proc/42800/task/44149/exe (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44149 ceph-osd ceph NOFD /proc/42800/task/44149/fd (opendir: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44150 osd_srv_a ceph cwd unknown /proc/42800/task/44150/cwd (readlink: Permission denied) 2024-04-22T06:40:17.694 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44150 osd_srv_a ceph rtd unknown /proc/42800/task/44150/root (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44150 osd_srv_a ceph txt unknown /proc/42800/task/44150/exe (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44150 osd_srv_a ceph NOFD /proc/42800/task/44150/fd (opendir: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44151 tp_osd_tp ceph cwd unknown /proc/42800/task/44151/cwd (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44151 tp_osd_tp ceph rtd unknown /proc/42800/task/44151/root (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44151 tp_osd_tp ceph txt unknown /proc/42800/task/44151/exe (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44151 tp_osd_tp ceph NOFD /proc/42800/task/44151/fd (opendir: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44152 tp_osd_tp ceph cwd unknown /proc/42800/task/44152/cwd (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44152 tp_osd_tp ceph rtd unknown /proc/42800/task/44152/root (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44152 tp_osd_tp ceph txt unknown /proc/42800/task/44152/exe (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44152 tp_osd_tp ceph NOFD /proc/42800/task/44152/fd (opendir: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44153 tp_osd_tp ceph cwd unknown /proc/42800/task/44153/cwd (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44153 tp_osd_tp ceph rtd unknown /proc/42800/task/44153/root (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44153 tp_osd_tp ceph txt unknown /proc/42800/task/44153/exe (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44153 tp_osd_tp ceph NOFD /proc/42800/task/44153/fd (opendir: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44154 tp_osd_tp ceph cwd unknown /proc/42800/task/44154/cwd (readlink: Permission denied) 2024-04-22T06:40:17.695 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44154 tp_osd_tp ceph rtd unknown /proc/42800/task/44154/root (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44154 tp_osd_tp ceph txt unknown /proc/42800/task/44154/exe (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44154 tp_osd_tp ceph NOFD /proc/42800/task/44154/fd (opendir: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44155 tp_osd_tp ceph cwd unknown /proc/42800/task/44155/cwd (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44155 tp_osd_tp ceph rtd unknown /proc/42800/task/44155/root (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44155 tp_osd_tp ceph txt unknown /proc/42800/task/44155/exe (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44155 tp_osd_tp ceph NOFD /proc/42800/task/44155/fd (opendir: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44156 tp_osd_tp ceph cwd unknown /proc/42800/task/44156/cwd (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44156 tp_osd_tp ceph rtd unknown /proc/42800/task/44156/root (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44156 tp_osd_tp ceph txt unknown /proc/42800/task/44156/exe (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44156 tp_osd_tp ceph NOFD /proc/42800/task/44156/fd (opendir: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44157 tp_osd_tp ceph cwd unknown /proc/42800/task/44157/cwd (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44157 tp_osd_tp ceph rtd unknown /proc/42800/task/44157/root (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44157 tp_osd_tp ceph txt unknown /proc/42800/task/44157/exe (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44157 tp_osd_tp ceph NOFD /proc/42800/task/44157/fd (opendir: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44158 tp_osd_tp ceph cwd unknown /proc/42800/task/44158/cwd (readlink: Permission denied) 2024-04-22T06:40:17.696 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44158 tp_osd_tp ceph rtd unknown /proc/42800/task/44158/root (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44158 tp_osd_tp ceph txt unknown /proc/42800/task/44158/exe (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44158 tp_osd_tp ceph NOFD /proc/42800/task/44158/fd (opendir: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44159 tp_osd_tp ceph cwd unknown /proc/42800/task/44159/cwd (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44159 tp_osd_tp ceph rtd unknown /proc/42800/task/44159/root (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44159 tp_osd_tp ceph txt unknown /proc/42800/task/44159/exe (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44159 tp_osd_tp ceph NOFD /proc/42800/task/44159/fd (opendir: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44160 tp_osd_tp ceph cwd unknown /proc/42800/task/44160/cwd (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44160 tp_osd_tp ceph rtd unknown /proc/42800/task/44160/root (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44160 tp_osd_tp ceph txt unknown /proc/42800/task/44160/exe (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44160 tp_osd_tp ceph NOFD /proc/42800/task/44160/fd (opendir: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44161 tp_osd_tp ceph cwd unknown /proc/42800/task/44161/cwd (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44161 tp_osd_tp ceph rtd unknown /proc/42800/task/44161/root (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44161 tp_osd_tp ceph txt unknown /proc/42800/task/44161/exe (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44161 tp_osd_tp ceph NOFD /proc/42800/task/44161/fd (opendir: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44162 tp_osd_tp ceph cwd unknown /proc/42800/task/44162/cwd (readlink: Permission denied) 2024-04-22T06:40:17.697 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44162 tp_osd_tp ceph rtd unknown /proc/42800/task/44162/root (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44162 tp_osd_tp ceph txt unknown /proc/42800/task/44162/exe (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44162 tp_osd_tp ceph NOFD /proc/42800/task/44162/fd (opendir: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44163 tp_osd_tp ceph cwd unknown /proc/42800/task/44163/cwd (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44163 tp_osd_tp ceph rtd unknown /proc/42800/task/44163/root (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44163 tp_osd_tp ceph txt unknown /proc/42800/task/44163/exe (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44163 tp_osd_tp ceph NOFD /proc/42800/task/44163/fd (opendir: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44164 tp_osd_tp ceph cwd unknown /proc/42800/task/44164/cwd (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44164 tp_osd_tp ceph rtd unknown /proc/42800/task/44164/root (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44164 tp_osd_tp ceph txt unknown /proc/42800/task/44164/exe (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44164 tp_osd_tp ceph NOFD /proc/42800/task/44164/fd (opendir: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44165 tp_osd_tp ceph cwd unknown /proc/42800/task/44165/cwd (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44165 tp_osd_tp ceph rtd unknown /proc/42800/task/44165/root (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44165 tp_osd_tp ceph txt unknown /proc/42800/task/44165/exe (readlink: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44165 tp_osd_tp ceph NOFD /proc/42800/task/44165/fd (opendir: Permission denied) 2024-04-22T06:40:17.698 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44166 tp_osd_tp ceph cwd unknown /proc/42800/task/44166/cwd (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44166 tp_osd_tp ceph rtd unknown /proc/42800/task/44166/root (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44166 tp_osd_tp ceph txt unknown /proc/42800/task/44166/exe (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44166 tp_osd_tp ceph NOFD /proc/42800/task/44166/fd (opendir: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44167 osd_srv_h ceph cwd unknown /proc/42800/task/44167/cwd (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44167 osd_srv_h ceph rtd unknown /proc/42800/task/44167/root (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44167 osd_srv_h ceph txt unknown /proc/42800/task/44167/exe (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 42800 44167 osd_srv_h ceph NOFD /proc/42800/task/44167/fd (opendir: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 44184 root cwd unknown /proc/44184/cwd (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 44184 root rtd unknown /proc/44184/root (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 44184 root txt unknown /proc/44184/exe (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 44184 root NOFD /proc/44184/fd (opendir: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 44204 root cwd unknown /proc/44204/cwd (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 44204 root rtd unknown /proc/44204/root (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 44204 root txt unknown /proc/44204/exe (readlink: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 44204 root NOFD /proc/44204/fd (opendir: Permission denied) 2024-04-22T06:40:17.699 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 46999 root cwd unknown /proc/46999/cwd (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 46999 root rtd unknown /proc/46999/root (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 46999 root txt unknown /proc/46999/exe (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 46999 root NOFD /proc/46999/fd (opendir: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 48731 root cwd unknown /proc/48731/cwd (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 48731 root rtd unknown /proc/48731/root (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 48731 root txt unknown /proc/48731/exe (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 48731 root NOFD /proc/48731/fd (opendir: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 49421 root cwd unknown /proc/49421/cwd (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 49421 root rtd unknown /proc/49421/root (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 49421 root txt unknown /proc/49421/exe (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 49421 root NOFD /proc/49421/fd (opendir: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 49487 root cwd unknown /proc/49487/cwd (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 49487 root rtd unknown /proc/49487/root (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 49487 root txt unknown /proc/49487/exe (readlink: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 49487 root NOFD /proc/49487/fd (opendir: Permission denied) 2024-04-22T06:40:17.700 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 49489 root cwd unknown /proc/49489/cwd (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 49489 root rtd unknown /proc/49489/root (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 49489 root txt unknown /proc/49489/exe (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 49489 root NOFD /proc/49489/fd (opendir: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 ceph cwd unknown /proc/49491/cwd (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 ceph rtd unknown /proc/49491/root (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 ceph txt unknown /proc/49491/exe (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 ceph NOFD /proc/49491/fd (opendir: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49492 log ceph cwd unknown /proc/49491/task/49492/cwd (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49492 log ceph rtd unknown /proc/49491/task/49492/root (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49492 log ceph txt unknown /proc/49491/task/49492/exe (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49492 log ceph NOFD /proc/49491/task/49492/fd (opendir: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49494 msgr-work ceph cwd unknown /proc/49491/task/49494/cwd (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49494 msgr-work ceph rtd unknown /proc/49491/task/49494/root (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49494 msgr-work ceph txt unknown /proc/49491/task/49494/exe (readlink: Permission denied) 2024-04-22T06:40:17.701 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49494 msgr-work ceph NOFD /proc/49491/task/49494/fd (opendir: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49495 msgr-work ceph cwd unknown /proc/49491/task/49495/cwd (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49495 msgr-work ceph rtd unknown /proc/49491/task/49495/root (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49495 msgr-work ceph txt unknown /proc/49491/task/49495/exe (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49495 msgr-work ceph NOFD /proc/49491/task/49495/fd (opendir: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49496 msgr-work ceph cwd unknown /proc/49491/task/49496/cwd (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49496 msgr-work ceph rtd unknown /proc/49491/task/49496/root (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49496 msgr-work ceph txt unknown /proc/49491/task/49496/exe (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49496 msgr-work ceph NOFD /proc/49491/task/49496/fd (opendir: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49500 service ceph cwd unknown /proc/49491/task/49500/cwd (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49500 service ceph rtd unknown /proc/49491/task/49500/root (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49500 service ceph txt unknown /proc/49491/task/49500/exe (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49500 service ceph NOFD /proc/49491/task/49500/fd (opendir: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49501 admin_soc ceph cwd unknown /proc/49491/task/49501/cwd (readlink: Permission denied) 2024-04-22T06:40:17.702 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49501 admin_soc ceph rtd unknown /proc/49491/task/49501/root (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49501 admin_soc ceph txt unknown /proc/49491/task/49501/exe (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49501 admin_soc ceph NOFD /proc/49491/task/49501/fd (opendir: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49643 signal_ha ceph cwd unknown /proc/49491/task/49643/cwd (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49643 signal_ha ceph rtd unknown /proc/49491/task/49643/root (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49643 signal_ha ceph txt unknown /proc/49491/task/49643/exe (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49643 signal_ha ceph NOFD /proc/49491/task/49643/fd (opendir: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49644 io_contex ceph cwd unknown /proc/49491/task/49644/cwd (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49644 io_contex ceph rtd unknown /proc/49491/task/49644/root (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49644 io_contex ceph txt unknown /proc/49491/task/49644/exe (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49644 io_contex ceph NOFD /proc/49491/task/49644/fd (opendir: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49645 io_contex ceph cwd unknown /proc/49491/task/49645/cwd (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49645 io_contex ceph rtd unknown /proc/49491/task/49645/root (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49645 io_contex ceph txt unknown /proc/49491/task/49645/exe (readlink: Permission denied) 2024-04-22T06:40:17.703 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49645 io_contex ceph NOFD /proc/49491/task/49645/fd (opendir: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49758 OpHistory ceph cwd unknown /proc/49491/task/49758/cwd (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49758 OpHistory ceph rtd unknown /proc/49491/task/49758/root (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49758 OpHistory ceph txt unknown /proc/49491/task/49758/exe (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49758 OpHistory ceph NOFD /proc/49491/task/49758/fd (opendir: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49759 ceph_time ceph cwd unknown /proc/49491/task/49759/cwd (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49759 ceph_time ceph rtd unknown /proc/49491/task/49759/root (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49759 ceph_time ceph txt unknown /proc/49491/task/49759/exe (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49759 ceph_time ceph NOFD /proc/49491/task/49759/fd (opendir: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49977 ceph-osd ceph cwd unknown /proc/49491/task/49977/cwd (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49977 ceph-osd ceph rtd unknown /proc/49491/task/49977/root (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49977 ceph-osd ceph txt unknown /proc/49491/task/49977/exe (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49977 ceph-osd ceph NOFD /proc/49491/task/49977/fd (opendir: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49978 safe_time ceph cwd unknown /proc/49491/task/49978/cwd (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49978 safe_time ceph rtd unknown /proc/49491/task/49978/root (readlink: Permission denied) 2024-04-22T06:40:17.704 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49978 safe_time ceph txt unknown /proc/49491/task/49978/exe (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49978 safe_time ceph NOFD /proc/49491/task/49978/fd (opendir: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49979 safe_time ceph cwd unknown /proc/49491/task/49979/cwd (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49979 safe_time ceph rtd unknown /proc/49491/task/49979/root (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49979 safe_time ceph txt unknown /proc/49491/task/49979/exe (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49979 safe_time ceph NOFD /proc/49491/task/49979/fd (opendir: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49980 safe_time ceph cwd unknown /proc/49491/task/49980/cwd (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49980 safe_time ceph rtd unknown /proc/49491/task/49980/root (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49980 safe_time ceph txt unknown /proc/49491/task/49980/exe (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49980 safe_time ceph NOFD /proc/49491/task/49980/fd (opendir: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49981 safe_time ceph cwd unknown /proc/49491/task/49981/cwd (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49981 safe_time ceph rtd unknown /proc/49491/task/49981/root (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49981 safe_time ceph txt unknown /proc/49491/task/49981/exe (readlink: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49981 safe_time ceph NOFD /proc/49491/task/49981/fd (opendir: Permission denied) 2024-04-22T06:40:17.705 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49982 fn_anonym ceph cwd unknown /proc/49491/task/49982/cwd (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49982 fn_anonym ceph rtd unknown /proc/49491/task/49982/root (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49982 fn_anonym ceph txt unknown /proc/49491/task/49982/exe (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49982 fn_anonym ceph NOFD /proc/49491/task/49982/fd (opendir: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49989 rocksdb:l ceph cwd unknown /proc/49491/task/49989/cwd (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49989 rocksdb:l ceph rtd unknown /proc/49491/task/49989/root (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49989 rocksdb:l ceph txt unknown /proc/49491/task/49989/exe (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49989 rocksdb:l ceph NOFD /proc/49491/task/49989/fd (opendir: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49990 rocksdb:l ceph cwd unknown /proc/49491/task/49990/cwd (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49990 rocksdb:l ceph rtd unknown /proc/49491/task/49990/root (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49990 rocksdb:l ceph txt unknown /proc/49491/task/49990/exe (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49990 rocksdb:l ceph NOFD /proc/49491/task/49990/fd (opendir: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49991 rocksdb:l ceph cwd unknown /proc/49491/task/49991/cwd (readlink: Permission denied) 2024-04-22T06:40:17.706 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49991 rocksdb:l ceph rtd unknown /proc/49491/task/49991/root (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49991 rocksdb:l ceph txt unknown /proc/49491/task/49991/exe (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49991 rocksdb:l ceph NOFD /proc/49491/task/49991/fd (opendir: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49992 rocksdb:h ceph cwd unknown /proc/49491/task/49992/cwd (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49992 rocksdb:h ceph rtd unknown /proc/49491/task/49992/root (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49992 rocksdb:h ceph txt unknown /proc/49491/task/49992/exe (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 49992 rocksdb:h ceph NOFD /proc/49491/task/49992/fd (opendir: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50385 bstore_ai ceph cwd unknown /proc/49491/task/50385/cwd (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50385 bstore_ai ceph rtd unknown /proc/49491/task/50385/root (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50385 bstore_ai ceph txt unknown /proc/49491/task/50385/exe (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50385 bstore_ai ceph NOFD /proc/49491/task/50385/fd (opendir: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50646 bstore_ai ceph cwd unknown /proc/49491/task/50646/cwd (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50646 bstore_ai ceph rtd unknown /proc/49491/task/50646/root (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50646 bstore_ai ceph txt unknown /proc/49491/task/50646/exe (readlink: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50646 bstore_ai ceph NOFD /proc/49491/task/50646/fd (opendir: Permission denied) 2024-04-22T06:40:17.707 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50829 ceph-osd ceph cwd unknown /proc/49491/task/50829/cwd (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50829 ceph-osd ceph rtd unknown /proc/49491/task/50829/root (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50829 ceph-osd ceph txt unknown /proc/49491/task/50829/exe (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50829 ceph-osd ceph NOFD /proc/49491/task/50829/fd (opendir: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50830 cfin ceph cwd unknown /proc/49491/task/50830/cwd (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50830 cfin ceph rtd unknown /proc/49491/task/50830/root (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50830 cfin ceph txt unknown /proc/49491/task/50830/exe (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50830 cfin ceph NOFD /proc/49491/task/50830/fd (opendir: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50831 bstore_kv ceph cwd unknown /proc/49491/task/50831/cwd (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50831 bstore_kv ceph rtd unknown /proc/49491/task/50831/root (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50831 bstore_kv ceph txt unknown /proc/49491/task/50831/exe (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50831 bstore_kv ceph NOFD /proc/49491/task/50831/fd (opendir: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50832 bstore_kv ceph cwd unknown /proc/49491/task/50832/cwd (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50832 bstore_kv ceph rtd unknown /proc/49491/task/50832/root (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50832 bstore_kv ceph txt unknown /proc/49491/task/50832/exe (readlink: Permission denied) 2024-04-22T06:40:17.708 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50832 bstore_kv ceph NOFD /proc/49491/task/50832/fd (opendir: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50833 bstore_me ceph cwd unknown /proc/49491/task/50833/cwd (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50833 bstore_me ceph rtd unknown /proc/49491/task/50833/root (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50833 bstore_me ceph txt unknown /proc/49491/task/50833/exe (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50833 bstore_me ceph NOFD /proc/49491/task/50833/fd (opendir: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50834 ms_dispat ceph cwd unknown /proc/49491/task/50834/cwd (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50834 ms_dispat ceph rtd unknown /proc/49491/task/50834/root (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50834 ms_dispat ceph txt unknown /proc/49491/task/50834/exe (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50834 ms_dispat ceph NOFD /proc/49491/task/50834/fd (opendir: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50835 ms_local ceph cwd unknown /proc/49491/task/50835/cwd (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50835 ms_local ceph rtd unknown /proc/49491/task/50835/root (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50835 ms_local ceph txt unknown /proc/49491/task/50835/exe (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50835 ms_local ceph NOFD /proc/49491/task/50835/fd (opendir: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50836 safe_time ceph cwd unknown /proc/49491/task/50836/cwd (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50836 safe_time ceph rtd unknown /proc/49491/task/50836/root (readlink: Permission denied) 2024-04-22T06:40:17.709 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50836 safe_time ceph txt unknown /proc/49491/task/50836/exe (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50836 safe_time ceph NOFD /proc/49491/task/50836/fd (opendir: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50837 safe_time ceph cwd unknown /proc/49491/task/50837/cwd (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50837 safe_time ceph rtd unknown /proc/49491/task/50837/root (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50837 safe_time ceph txt unknown /proc/49491/task/50837/exe (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50837 safe_time ceph NOFD /proc/49491/task/50837/fd (opendir: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50838 ms_dispat ceph cwd unknown /proc/49491/task/50838/cwd (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50838 ms_dispat ceph rtd unknown /proc/49491/task/50838/root (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50838 ms_dispat ceph txt unknown /proc/49491/task/50838/exe (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50838 ms_dispat ceph NOFD /proc/49491/task/50838/fd (opendir: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50839 ms_local ceph cwd unknown /proc/49491/task/50839/cwd (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50839 ms_local ceph rtd unknown /proc/49491/task/50839/root (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50839 ms_local ceph txt unknown /proc/49491/task/50839/exe (readlink: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50839 ms_local ceph NOFD /proc/49491/task/50839/fd (opendir: Permission denied) 2024-04-22T06:40:17.710 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50840 ms_dispat ceph cwd unknown /proc/49491/task/50840/cwd (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50840 ms_dispat ceph rtd unknown /proc/49491/task/50840/root (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50840 ms_dispat ceph txt unknown /proc/49491/task/50840/exe (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50840 ms_dispat ceph NOFD /proc/49491/task/50840/fd (opendir: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50841 ms_local ceph cwd unknown /proc/49491/task/50841/cwd (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50841 ms_local ceph rtd unknown /proc/49491/task/50841/root (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50841 ms_local ceph txt unknown /proc/49491/task/50841/exe (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50841 ms_local ceph NOFD /proc/49491/task/50841/fd (opendir: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50842 ms_dispat ceph cwd unknown /proc/49491/task/50842/cwd (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50842 ms_dispat ceph rtd unknown /proc/49491/task/50842/root (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50842 ms_dispat ceph txt unknown /proc/49491/task/50842/exe (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50842 ms_dispat ceph NOFD /proc/49491/task/50842/fd (opendir: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50843 ms_local ceph cwd unknown /proc/49491/task/50843/cwd (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50843 ms_local ceph rtd unknown /proc/49491/task/50843/root (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50843 ms_local ceph txt unknown /proc/49491/task/50843/exe (readlink: Permission denied) 2024-04-22T06:40:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50843 ms_local ceph NOFD /proc/49491/task/50843/fd (opendir: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50844 ms_dispat ceph cwd unknown /proc/49491/task/50844/cwd (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50844 ms_dispat ceph rtd unknown /proc/49491/task/50844/root (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50844 ms_dispat ceph txt unknown /proc/49491/task/50844/exe (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50844 ms_dispat ceph NOFD /proc/49491/task/50844/fd (opendir: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50845 ms_local ceph cwd unknown /proc/49491/task/50845/cwd (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50845 ms_local ceph rtd unknown /proc/49491/task/50845/root (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50845 ms_local ceph txt unknown /proc/49491/task/50845/exe (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50845 ms_local ceph NOFD /proc/49491/task/50845/fd (opendir: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50846 ms_dispat ceph cwd unknown /proc/49491/task/50846/cwd (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50846 ms_dispat ceph rtd unknown /proc/49491/task/50846/root (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50846 ms_dispat ceph txt unknown /proc/49491/task/50846/exe (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50846 ms_dispat ceph NOFD /proc/49491/task/50846/fd (opendir: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50847 ms_local ceph cwd unknown /proc/49491/task/50847/cwd (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50847 ms_local ceph rtd unknown /proc/49491/task/50847/root (readlink: Permission denied) 2024-04-22T06:40:17.712 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50847 ms_local ceph txt unknown /proc/49491/task/50847/exe (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50847 ms_local ceph NOFD /proc/49491/task/50847/fd (opendir: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50848 ms_dispat ceph cwd unknown /proc/49491/task/50848/cwd (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50848 ms_dispat ceph rtd unknown /proc/49491/task/50848/root (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50848 ms_dispat ceph txt unknown /proc/49491/task/50848/exe (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50848 ms_dispat ceph NOFD /proc/49491/task/50848/fd (opendir: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50849 ms_local ceph cwd unknown /proc/49491/task/50849/cwd (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50849 ms_local ceph rtd unknown /proc/49491/task/50849/root (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50849 ms_local ceph txt unknown /proc/49491/task/50849/exe (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50849 ms_local ceph NOFD /proc/49491/task/50849/fd (opendir: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50850 fn_anonym ceph cwd unknown /proc/49491/task/50850/cwd (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50850 fn_anonym ceph rtd unknown /proc/49491/task/50850/root (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50850 fn_anonym ceph txt unknown /proc/49491/task/50850/exe (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50850 fn_anonym ceph NOFD /proc/49491/task/50850/fd (opendir: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50851 finisher ceph cwd unknown /proc/49491/task/50851/cwd (readlink: Permission denied) 2024-04-22T06:40:17.713 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50851 finisher ceph rtd unknown /proc/49491/task/50851/root (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50851 finisher ceph txt unknown /proc/49491/task/50851/exe (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50851 finisher ceph NOFD /proc/49491/task/50851/fd (opendir: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50852 safe_time ceph cwd unknown /proc/49491/task/50852/cwd (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50852 safe_time ceph rtd unknown /proc/49491/task/50852/root (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50852 safe_time ceph txt unknown /proc/49491/task/50852/exe (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50852 safe_time ceph NOFD /proc/49491/task/50852/fd (opendir: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50853 safe_time ceph cwd unknown /proc/49491/task/50853/cwd (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50853 safe_time ceph rtd unknown /proc/49491/task/50853/root (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50853 safe_time ceph txt unknown /proc/49491/task/50853/exe (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50853 safe_time ceph NOFD /proc/49491/task/50853/fd (opendir: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50854 ceph-osd ceph cwd unknown /proc/49491/task/50854/cwd (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50854 ceph-osd ceph rtd unknown /proc/49491/task/50854/root (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50854 ceph-osd ceph txt unknown /proc/49491/task/50854/exe (readlink: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50854 ceph-osd ceph NOFD /proc/49491/task/50854/fd (opendir: Permission denied) 2024-04-22T06:40:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50855 osd_srv_a ceph cwd unknown /proc/49491/task/50855/cwd (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50855 osd_srv_a ceph rtd unknown /proc/49491/task/50855/root (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50855 osd_srv_a ceph txt unknown /proc/49491/task/50855/exe (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50855 osd_srv_a ceph NOFD /proc/49491/task/50855/fd (opendir: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50856 tp_osd_tp ceph cwd unknown /proc/49491/task/50856/cwd (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50856 tp_osd_tp ceph rtd unknown /proc/49491/task/50856/root (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50856 tp_osd_tp ceph txt unknown /proc/49491/task/50856/exe (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50856 tp_osd_tp ceph NOFD /proc/49491/task/50856/fd (opendir: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50857 tp_osd_tp ceph cwd unknown /proc/49491/task/50857/cwd (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50857 tp_osd_tp ceph rtd unknown /proc/49491/task/50857/root (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50857 tp_osd_tp ceph txt unknown /proc/49491/task/50857/exe (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50857 tp_osd_tp ceph NOFD /proc/49491/task/50857/fd (opendir: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50858 tp_osd_tp ceph cwd unknown /proc/49491/task/50858/cwd (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50858 tp_osd_tp ceph rtd unknown /proc/49491/task/50858/root (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50858 tp_osd_tp ceph txt unknown /proc/49491/task/50858/exe (readlink: Permission denied) 2024-04-22T06:40:17.715 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50858 tp_osd_tp ceph NOFD /proc/49491/task/50858/fd (opendir: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50859 tp_osd_tp ceph cwd unknown /proc/49491/task/50859/cwd (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50859 tp_osd_tp ceph rtd unknown /proc/49491/task/50859/root (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50859 tp_osd_tp ceph txt unknown /proc/49491/task/50859/exe (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50859 tp_osd_tp ceph NOFD /proc/49491/task/50859/fd (opendir: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50860 tp_osd_tp ceph cwd unknown /proc/49491/task/50860/cwd (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50860 tp_osd_tp ceph rtd unknown /proc/49491/task/50860/root (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50860 tp_osd_tp ceph txt unknown /proc/49491/task/50860/exe (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50860 tp_osd_tp ceph NOFD /proc/49491/task/50860/fd (opendir: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50861 tp_osd_tp ceph cwd unknown /proc/49491/task/50861/cwd (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50861 tp_osd_tp ceph rtd unknown /proc/49491/task/50861/root (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50861 tp_osd_tp ceph txt unknown /proc/49491/task/50861/exe (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50861 tp_osd_tp ceph NOFD /proc/49491/task/50861/fd (opendir: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50862 tp_osd_tp ceph cwd unknown /proc/49491/task/50862/cwd (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50862 tp_osd_tp ceph rtd unknown /proc/49491/task/50862/root (readlink: Permission denied) 2024-04-22T06:40:17.716 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50862 tp_osd_tp ceph txt unknown /proc/49491/task/50862/exe (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50862 tp_osd_tp ceph NOFD /proc/49491/task/50862/fd (opendir: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50863 tp_osd_tp ceph cwd unknown /proc/49491/task/50863/cwd (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50863 tp_osd_tp ceph rtd unknown /proc/49491/task/50863/root (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50863 tp_osd_tp ceph txt unknown /proc/49491/task/50863/exe (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50863 tp_osd_tp ceph NOFD /proc/49491/task/50863/fd (opendir: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50864 tp_osd_tp ceph cwd unknown /proc/49491/task/50864/cwd (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50864 tp_osd_tp ceph rtd unknown /proc/49491/task/50864/root (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50864 tp_osd_tp ceph txt unknown /proc/49491/task/50864/exe (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50864 tp_osd_tp ceph NOFD /proc/49491/task/50864/fd (opendir: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50865 tp_osd_tp ceph cwd unknown /proc/49491/task/50865/cwd (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50865 tp_osd_tp ceph rtd unknown /proc/49491/task/50865/root (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50865 tp_osd_tp ceph txt unknown /proc/49491/task/50865/exe (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50865 tp_osd_tp ceph NOFD /proc/49491/task/50865/fd (opendir: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50866 tp_osd_tp ceph cwd unknown /proc/49491/task/50866/cwd (readlink: Permission denied) 2024-04-22T06:40:17.717 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50866 tp_osd_tp ceph rtd unknown /proc/49491/task/50866/root (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50866 tp_osd_tp ceph txt unknown /proc/49491/task/50866/exe (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50866 tp_osd_tp ceph NOFD /proc/49491/task/50866/fd (opendir: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50867 tp_osd_tp ceph cwd unknown /proc/49491/task/50867/cwd (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50867 tp_osd_tp ceph rtd unknown /proc/49491/task/50867/root (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50867 tp_osd_tp ceph txt unknown /proc/49491/task/50867/exe (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50867 tp_osd_tp ceph NOFD /proc/49491/task/50867/fd (opendir: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50868 tp_osd_tp ceph cwd unknown /proc/49491/task/50868/cwd (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50868 tp_osd_tp ceph rtd unknown /proc/49491/task/50868/root (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50868 tp_osd_tp ceph txt unknown /proc/49491/task/50868/exe (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50868 tp_osd_tp ceph NOFD /proc/49491/task/50868/fd (opendir: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50869 tp_osd_tp ceph cwd unknown /proc/49491/task/50869/cwd (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50869 tp_osd_tp ceph rtd unknown /proc/49491/task/50869/root (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50869 tp_osd_tp ceph txt unknown /proc/49491/task/50869/exe (readlink: Permission denied) 2024-04-22T06:40:17.718 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50869 tp_osd_tp ceph NOFD /proc/49491/task/50869/fd (opendir: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50870 tp_osd_tp ceph cwd unknown /proc/49491/task/50870/cwd (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50870 tp_osd_tp ceph rtd unknown /proc/49491/task/50870/root (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50870 tp_osd_tp ceph txt unknown /proc/49491/task/50870/exe (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50870 tp_osd_tp ceph NOFD /proc/49491/task/50870/fd (opendir: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50871 tp_osd_tp ceph cwd unknown /proc/49491/task/50871/cwd (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50871 tp_osd_tp ceph rtd unknown /proc/49491/task/50871/root (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50871 tp_osd_tp ceph txt unknown /proc/49491/task/50871/exe (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50871 tp_osd_tp ceph NOFD /proc/49491/task/50871/fd (opendir: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50872 osd_srv_h ceph cwd unknown /proc/49491/task/50872/cwd (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50872 osd_srv_h ceph rtd unknown /proc/49491/task/50872/root (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50872 osd_srv_h ceph txt unknown /proc/49491/task/50872/exe (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 49491 50872 osd_srv_h ceph NOFD /proc/49491/task/50872/fd (opendir: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 50886 root cwd unknown /proc/50886/cwd (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 50886 root rtd unknown /proc/50886/root (readlink: Permission denied) 2024-04-22T06:40:17.719 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 50886 root txt unknown /proc/50886/exe (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 50886 root NOFD /proc/50886/fd (opendir: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 50887 root cwd unknown /proc/50887/cwd (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 50887 root rtd unknown /proc/50887/root (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 50887 root txt unknown /proc/50887/exe (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 50887 root NOFD /proc/50887/fd (opendir: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 50917 root cwd unknown /proc/50917/cwd (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 50917 root rtd unknown /proc/50917/root (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 50917 root txt unknown /proc/50917/exe (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 50917 root NOFD /proc/50917/fd (opendir: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 56494 root cwd unknown /proc/56494/cwd (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 56494 root rtd unknown /proc/56494/root (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 56494 root txt unknown /proc/56494/exe (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 56494 root NOFD /proc/56494/fd (opendir: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 56496 root cwd unknown /proc/56496/cwd (readlink: Permission denied) 2024-04-22T06:40:17.720 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 56496 root rtd unknown /proc/56496/root (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 56496 root txt unknown /proc/56496/exe (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 56496 root NOFD /proc/56496/fd (opendir: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 ceph cwd unknown /proc/56498/cwd (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 ceph rtd unknown /proc/56498/root (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 ceph txt unknown /proc/56498/exe (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 ceph NOFD /proc/56498/fd (opendir: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56499 log ceph cwd unknown /proc/56498/task/56499/cwd (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56499 log ceph rtd unknown /proc/56498/task/56499/root (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56499 log ceph txt unknown /proc/56498/task/56499/exe (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56499 log ceph NOFD /proc/56498/task/56499/fd (opendir: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56501 msgr-work ceph cwd unknown /proc/56498/task/56501/cwd (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56501 msgr-work ceph rtd unknown /proc/56498/task/56501/root (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56501 msgr-work ceph txt unknown /proc/56498/task/56501/exe (readlink: Permission denied) 2024-04-22T06:40:17.721 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56501 msgr-work ceph NOFD /proc/56498/task/56501/fd (opendir: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56502 msgr-work ceph cwd unknown /proc/56498/task/56502/cwd (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56502 msgr-work ceph rtd unknown /proc/56498/task/56502/root (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56502 msgr-work ceph txt unknown /proc/56498/task/56502/exe (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56502 msgr-work ceph NOFD /proc/56498/task/56502/fd (opendir: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56503 msgr-work ceph cwd unknown /proc/56498/task/56503/cwd (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56503 msgr-work ceph rtd unknown /proc/56498/task/56503/root (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56503 msgr-work ceph txt unknown /proc/56498/task/56503/exe (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56503 msgr-work ceph NOFD /proc/56498/task/56503/fd (opendir: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56507 service ceph cwd unknown /proc/56498/task/56507/cwd (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56507 service ceph rtd unknown /proc/56498/task/56507/root (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56507 service ceph txt unknown /proc/56498/task/56507/exe (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56507 service ceph NOFD /proc/56498/task/56507/fd (opendir: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56508 admin_soc ceph cwd unknown /proc/56498/task/56508/cwd (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56508 admin_soc ceph rtd unknown /proc/56498/task/56508/root (readlink: Permission denied) 2024-04-22T06:40:17.722 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56508 admin_soc ceph txt unknown /proc/56498/task/56508/exe (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56508 admin_soc ceph NOFD /proc/56498/task/56508/fd (opendir: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56650 signal_ha ceph cwd unknown /proc/56498/task/56650/cwd (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56650 signal_ha ceph rtd unknown /proc/56498/task/56650/root (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56650 signal_ha ceph txt unknown /proc/56498/task/56650/exe (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56650 signal_ha ceph NOFD /proc/56498/task/56650/fd (opendir: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56651 io_contex ceph cwd unknown /proc/56498/task/56651/cwd (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56651 io_contex ceph rtd unknown /proc/56498/task/56651/root (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56651 io_contex ceph txt unknown /proc/56498/task/56651/exe (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56651 io_contex ceph NOFD /proc/56498/task/56651/fd (opendir: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56652 io_contex ceph cwd unknown /proc/56498/task/56652/cwd (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56652 io_contex ceph rtd unknown /proc/56498/task/56652/root (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56652 io_contex ceph txt unknown /proc/56498/task/56652/exe (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56652 io_contex ceph NOFD /proc/56498/task/56652/fd (opendir: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56766 OpHistory ceph cwd unknown /proc/56498/task/56766/cwd (readlink: Permission denied) 2024-04-22T06:40:17.723 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56766 OpHistory ceph rtd unknown /proc/56498/task/56766/root (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56766 OpHistory ceph txt unknown /proc/56498/task/56766/exe (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56766 OpHistory ceph NOFD /proc/56498/task/56766/fd (opendir: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56767 ceph_time ceph cwd unknown /proc/56498/task/56767/cwd (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56767 ceph_time ceph rtd unknown /proc/56498/task/56767/root (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56767 ceph_time ceph txt unknown /proc/56498/task/56767/exe (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56767 ceph_time ceph NOFD /proc/56498/task/56767/fd (opendir: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56789 ceph-osd ceph cwd unknown /proc/56498/task/56789/cwd (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56789 ceph-osd ceph rtd unknown /proc/56498/task/56789/root (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56789 ceph-osd ceph txt unknown /proc/56498/task/56789/exe (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56789 ceph-osd ceph NOFD /proc/56498/task/56789/fd (opendir: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56801 ceph-osd ceph cwd unknown /proc/56498/task/56801/cwd (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56801 ceph-osd ceph rtd unknown /proc/56498/task/56801/root (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56801 ceph-osd ceph txt unknown /proc/56498/task/56801/exe (readlink: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56801 ceph-osd ceph NOFD /proc/56498/task/56801/fd (opendir: Permission denied) 2024-04-22T06:40:17.724 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56827 ceph-osd ceph cwd unknown /proc/56498/task/56827/cwd (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56827 ceph-osd ceph rtd unknown /proc/56498/task/56827/root (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56827 ceph-osd ceph txt unknown /proc/56498/task/56827/exe (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56827 ceph-osd ceph NOFD /proc/56498/task/56827/fd (opendir: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56852 ceph-osd ceph cwd unknown /proc/56498/task/56852/cwd (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56852 ceph-osd ceph rtd unknown /proc/56498/task/56852/root (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56852 ceph-osd ceph txt unknown /proc/56498/task/56852/exe (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56852 ceph-osd ceph NOFD /proc/56498/task/56852/fd (opendir: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56875 ceph-osd ceph cwd unknown /proc/56498/task/56875/cwd (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56875 ceph-osd ceph rtd unknown /proc/56498/task/56875/root (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56875 ceph-osd ceph txt unknown /proc/56498/task/56875/exe (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56875 ceph-osd ceph NOFD /proc/56498/task/56875/fd (opendir: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56885 ceph-osd ceph cwd unknown /proc/56498/task/56885/cwd (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56885 ceph-osd ceph rtd unknown /proc/56498/task/56885/root (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56885 ceph-osd ceph txt unknown /proc/56498/task/56885/exe (readlink: Permission denied) 2024-04-22T06:40:17.725 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56885 ceph-osd ceph NOFD /proc/56498/task/56885/fd (opendir: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56942 ceph-osd ceph cwd unknown /proc/56498/task/56942/cwd (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56942 ceph-osd ceph rtd unknown /proc/56498/task/56942/root (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56942 ceph-osd ceph txt unknown /proc/56498/task/56942/exe (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56942 ceph-osd ceph NOFD /proc/56498/task/56942/fd (opendir: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56986 ceph-osd ceph cwd unknown /proc/56498/task/56986/cwd (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56986 ceph-osd ceph rtd unknown /proc/56498/task/56986/root (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56986 ceph-osd ceph txt unknown /proc/56498/task/56986/exe (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56986 ceph-osd ceph NOFD /proc/56498/task/56986/fd (opendir: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56988 ceph-osd ceph cwd unknown /proc/56498/task/56988/cwd (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56988 ceph-osd ceph rtd unknown /proc/56498/task/56988/root (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56988 ceph-osd ceph txt unknown /proc/56498/task/56988/exe (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56988 ceph-osd ceph NOFD /proc/56498/task/56988/fd (opendir: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56989 safe_time ceph cwd unknown /proc/56498/task/56989/cwd (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56989 safe_time ceph rtd unknown /proc/56498/task/56989/root (readlink: Permission denied) 2024-04-22T06:40:17.726 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56989 safe_time ceph txt unknown /proc/56498/task/56989/exe (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56989 safe_time ceph NOFD /proc/56498/task/56989/fd (opendir: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56990 safe_time ceph cwd unknown /proc/56498/task/56990/cwd (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56990 safe_time ceph rtd unknown /proc/56498/task/56990/root (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56990 safe_time ceph txt unknown /proc/56498/task/56990/exe (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56990 safe_time ceph NOFD /proc/56498/task/56990/fd (opendir: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56991 safe_time ceph cwd unknown /proc/56498/task/56991/cwd (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56991 safe_time ceph rtd unknown /proc/56498/task/56991/root (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56991 safe_time ceph txt unknown /proc/56498/task/56991/exe (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56991 safe_time ceph NOFD /proc/56498/task/56991/fd (opendir: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56992 safe_time ceph cwd unknown /proc/56498/task/56992/cwd (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56992 safe_time ceph rtd unknown /proc/56498/task/56992/root (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56992 safe_time ceph txt unknown /proc/56498/task/56992/exe (readlink: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56992 safe_time ceph NOFD /proc/56498/task/56992/fd (opendir: Permission denied) 2024-04-22T06:40:17.727 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56993 fn_anonym ceph cwd unknown /proc/56498/task/56993/cwd (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56993 fn_anonym ceph rtd unknown /proc/56498/task/56993/root (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56993 fn_anonym ceph txt unknown /proc/56498/task/56993/exe (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 56993 fn_anonym ceph NOFD /proc/56498/task/56993/fd (opendir: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57007 rocksdb:l ceph cwd unknown /proc/56498/task/57007/cwd (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57007 rocksdb:l ceph rtd unknown /proc/56498/task/57007/root (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57007 rocksdb:l ceph txt unknown /proc/56498/task/57007/exe (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57007 rocksdb:l ceph NOFD /proc/56498/task/57007/fd (opendir: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57008 rocksdb:l ceph cwd unknown /proc/56498/task/57008/cwd (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57008 rocksdb:l ceph rtd unknown /proc/56498/task/57008/root (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57008 rocksdb:l ceph txt unknown /proc/56498/task/57008/exe (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57008 rocksdb:l ceph NOFD /proc/56498/task/57008/fd (opendir: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57009 rocksdb:l ceph cwd unknown /proc/56498/task/57009/cwd (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57009 rocksdb:l ceph rtd unknown /proc/56498/task/57009/root (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57009 rocksdb:l ceph txt unknown /proc/56498/task/57009/exe (readlink: Permission denied) 2024-04-22T06:40:17.728 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57009 rocksdb:l ceph NOFD /proc/56498/task/57009/fd (opendir: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57010 rocksdb:h ceph cwd unknown /proc/56498/task/57010/cwd (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57010 rocksdb:h ceph rtd unknown /proc/56498/task/57010/root (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57010 rocksdb:h ceph txt unknown /proc/56498/task/57010/exe (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57010 rocksdb:h ceph NOFD /proc/56498/task/57010/fd (opendir: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57420 bstore_ai ceph cwd unknown /proc/56498/task/57420/cwd (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57420 bstore_ai ceph rtd unknown /proc/56498/task/57420/root (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57420 bstore_ai ceph txt unknown /proc/56498/task/57420/exe (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57420 bstore_ai ceph NOFD /proc/56498/task/57420/fd (opendir: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57682 bstore_ai ceph cwd unknown /proc/56498/task/57682/cwd (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57682 bstore_ai ceph rtd unknown /proc/56498/task/57682/root (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57682 bstore_ai ceph txt unknown /proc/56498/task/57682/exe (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57682 bstore_ai ceph NOFD /proc/56498/task/57682/fd (opendir: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57864 ceph-osd ceph cwd unknown /proc/56498/task/57864/cwd (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57864 ceph-osd ceph rtd unknown /proc/56498/task/57864/root (readlink: Permission denied) 2024-04-22T06:40:17.729 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57864 ceph-osd ceph txt unknown /proc/56498/task/57864/exe (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57864 ceph-osd ceph NOFD /proc/56498/task/57864/fd (opendir: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57865 cfin ceph cwd unknown /proc/56498/task/57865/cwd (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57865 cfin ceph rtd unknown /proc/56498/task/57865/root (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57865 cfin ceph txt unknown /proc/56498/task/57865/exe (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57865 cfin ceph NOFD /proc/56498/task/57865/fd (opendir: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57866 bstore_kv ceph cwd unknown /proc/56498/task/57866/cwd (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57866 bstore_kv ceph rtd unknown /proc/56498/task/57866/root (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57866 bstore_kv ceph txt unknown /proc/56498/task/57866/exe (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57866 bstore_kv ceph NOFD /proc/56498/task/57866/fd (opendir: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57867 bstore_kv ceph cwd unknown /proc/56498/task/57867/cwd (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57867 bstore_kv ceph rtd unknown /proc/56498/task/57867/root (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57867 bstore_kv ceph txt unknown /proc/56498/task/57867/exe (readlink: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57867 bstore_kv ceph NOFD /proc/56498/task/57867/fd (opendir: Permission denied) 2024-04-22T06:40:17.730 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57868 bstore_me ceph cwd unknown /proc/56498/task/57868/cwd (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57868 bstore_me ceph rtd unknown /proc/56498/task/57868/root (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57868 bstore_me ceph txt unknown /proc/56498/task/57868/exe (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57868 bstore_me ceph NOFD /proc/56498/task/57868/fd (opendir: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57869 ms_dispat ceph cwd unknown /proc/56498/task/57869/cwd (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57869 ms_dispat ceph rtd unknown /proc/56498/task/57869/root (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57869 ms_dispat ceph txt unknown /proc/56498/task/57869/exe (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57869 ms_dispat ceph NOFD /proc/56498/task/57869/fd (opendir: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57870 ms_local ceph cwd unknown /proc/56498/task/57870/cwd (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57870 ms_local ceph rtd unknown /proc/56498/task/57870/root (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57870 ms_local ceph txt unknown /proc/56498/task/57870/exe (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57870 ms_local ceph NOFD /proc/56498/task/57870/fd (opendir: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57871 safe_time ceph cwd unknown /proc/56498/task/57871/cwd (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57871 safe_time ceph rtd unknown /proc/56498/task/57871/root (readlink: Permission denied) 2024-04-22T06:40:17.731 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57871 safe_time ceph txt unknown /proc/56498/task/57871/exe (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57871 safe_time ceph NOFD /proc/56498/task/57871/fd (opendir: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57872 safe_time ceph cwd unknown /proc/56498/task/57872/cwd (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57872 safe_time ceph rtd unknown /proc/56498/task/57872/root (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57872 safe_time ceph txt unknown /proc/56498/task/57872/exe (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57872 safe_time ceph NOFD /proc/56498/task/57872/fd (opendir: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57873 ms_dispat ceph cwd unknown /proc/56498/task/57873/cwd (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57873 ms_dispat ceph rtd unknown /proc/56498/task/57873/root (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57873 ms_dispat ceph txt unknown /proc/56498/task/57873/exe (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57873 ms_dispat ceph NOFD /proc/56498/task/57873/fd (opendir: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57874 ms_local ceph cwd unknown /proc/56498/task/57874/cwd (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57874 ms_local ceph rtd unknown /proc/56498/task/57874/root (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57874 ms_local ceph txt unknown /proc/56498/task/57874/exe (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57874 ms_local ceph NOFD /proc/56498/task/57874/fd (opendir: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57875 ms_dispat ceph cwd unknown /proc/56498/task/57875/cwd (readlink: Permission denied) 2024-04-22T06:40:17.732 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57875 ms_dispat ceph rtd unknown /proc/56498/task/57875/root (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57875 ms_dispat ceph txt unknown /proc/56498/task/57875/exe (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57875 ms_dispat ceph NOFD /proc/56498/task/57875/fd (opendir: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57876 ms_local ceph cwd unknown /proc/56498/task/57876/cwd (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57876 ms_local ceph rtd unknown /proc/56498/task/57876/root (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57876 ms_local ceph txt unknown /proc/56498/task/57876/exe (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57876 ms_local ceph NOFD /proc/56498/task/57876/fd (opendir: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57877 ms_dispat ceph cwd unknown /proc/56498/task/57877/cwd (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57877 ms_dispat ceph rtd unknown /proc/56498/task/57877/root (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57877 ms_dispat ceph txt unknown /proc/56498/task/57877/exe (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57877 ms_dispat ceph NOFD /proc/56498/task/57877/fd (opendir: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57878 ms_local ceph cwd unknown /proc/56498/task/57878/cwd (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57878 ms_local ceph rtd unknown /proc/56498/task/57878/root (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57878 ms_local ceph txt unknown /proc/56498/task/57878/exe (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57878 ms_local ceph NOFD /proc/56498/task/57878/fd (opendir: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57879 ms_dispat ceph cwd unknown /proc/56498/task/57879/cwd (readlink: Permission denied) 2024-04-22T06:40:17.733 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57879 ms_dispat ceph rtd unknown /proc/56498/task/57879/root (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57879 ms_dispat ceph txt unknown /proc/56498/task/57879/exe (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57879 ms_dispat ceph NOFD /proc/56498/task/57879/fd (opendir: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57880 ms_local ceph cwd unknown /proc/56498/task/57880/cwd (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57880 ms_local ceph rtd unknown /proc/56498/task/57880/root (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57880 ms_local ceph txt unknown /proc/56498/task/57880/exe (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57880 ms_local ceph NOFD /proc/56498/task/57880/fd (opendir: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57881 ms_dispat ceph cwd unknown /proc/56498/task/57881/cwd (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57881 ms_dispat ceph rtd unknown /proc/56498/task/57881/root (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57881 ms_dispat ceph txt unknown /proc/56498/task/57881/exe (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57881 ms_dispat ceph NOFD /proc/56498/task/57881/fd (opendir: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57882 ms_local ceph cwd unknown /proc/56498/task/57882/cwd (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57882 ms_local ceph rtd unknown /proc/56498/task/57882/root (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57882 ms_local ceph txt unknown /proc/56498/task/57882/exe (readlink: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57882 ms_local ceph NOFD /proc/56498/task/57882/fd (opendir: Permission denied) 2024-04-22T06:40:17.734 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57883 ms_dispat ceph cwd unknown /proc/56498/task/57883/cwd (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57883 ms_dispat ceph rtd unknown /proc/56498/task/57883/root (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57883 ms_dispat ceph txt unknown /proc/56498/task/57883/exe (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57883 ms_dispat ceph NOFD /proc/56498/task/57883/fd (opendir: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57884 ms_local ceph cwd unknown /proc/56498/task/57884/cwd (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57884 ms_local ceph rtd unknown /proc/56498/task/57884/root (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57884 ms_local ceph txt unknown /proc/56498/task/57884/exe (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57884 ms_local ceph NOFD /proc/56498/task/57884/fd (opendir: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57885 fn_anonym ceph cwd unknown /proc/56498/task/57885/cwd (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57885 fn_anonym ceph rtd unknown /proc/56498/task/57885/root (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57885 fn_anonym ceph txt unknown /proc/56498/task/57885/exe (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57885 fn_anonym ceph NOFD /proc/56498/task/57885/fd (opendir: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57886 finisher ceph cwd unknown /proc/56498/task/57886/cwd (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57886 finisher ceph rtd unknown /proc/56498/task/57886/root (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57886 finisher ceph txt unknown /proc/56498/task/57886/exe (readlink: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57886 finisher ceph NOFD /proc/56498/task/57886/fd (opendir: Permission denied) 2024-04-22T06:40:17.735 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57887 safe_time ceph cwd unknown /proc/56498/task/57887/cwd (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57887 safe_time ceph rtd unknown /proc/56498/task/57887/root (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57887 safe_time ceph txt unknown /proc/56498/task/57887/exe (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57887 safe_time ceph NOFD /proc/56498/task/57887/fd (opendir: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57888 safe_time ceph cwd unknown /proc/56498/task/57888/cwd (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57888 safe_time ceph rtd unknown /proc/56498/task/57888/root (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57888 safe_time ceph txt unknown /proc/56498/task/57888/exe (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57888 safe_time ceph NOFD /proc/56498/task/57888/fd (opendir: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57889 ceph-osd ceph cwd unknown /proc/56498/task/57889/cwd (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57889 ceph-osd ceph rtd unknown /proc/56498/task/57889/root (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57889 ceph-osd ceph txt unknown /proc/56498/task/57889/exe (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57889 ceph-osd ceph NOFD /proc/56498/task/57889/fd (opendir: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57890 osd_srv_a ceph cwd unknown /proc/56498/task/57890/cwd (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57890 osd_srv_a ceph rtd unknown /proc/56498/task/57890/root (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57890 osd_srv_a ceph txt unknown /proc/56498/task/57890/exe (readlink: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57890 osd_srv_a ceph NOFD /proc/56498/task/57890/fd (opendir: Permission denied) 2024-04-22T06:40:17.736 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57891 tp_osd_tp ceph cwd unknown /proc/56498/task/57891/cwd (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57891 tp_osd_tp ceph rtd unknown /proc/56498/task/57891/root (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57891 tp_osd_tp ceph txt unknown /proc/56498/task/57891/exe (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57891 tp_osd_tp ceph NOFD /proc/56498/task/57891/fd (opendir: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57892 tp_osd_tp ceph cwd unknown /proc/56498/task/57892/cwd (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57892 tp_osd_tp ceph rtd unknown /proc/56498/task/57892/root (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57892 tp_osd_tp ceph txt unknown /proc/56498/task/57892/exe (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57892 tp_osd_tp ceph NOFD /proc/56498/task/57892/fd (opendir: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57893 tp_osd_tp ceph cwd unknown /proc/56498/task/57893/cwd (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57893 tp_osd_tp ceph rtd unknown /proc/56498/task/57893/root (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57893 tp_osd_tp ceph txt unknown /proc/56498/task/57893/exe (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57893 tp_osd_tp ceph NOFD /proc/56498/task/57893/fd (opendir: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57894 tp_osd_tp ceph cwd unknown /proc/56498/task/57894/cwd (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57894 tp_osd_tp ceph rtd unknown /proc/56498/task/57894/root (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57894 tp_osd_tp ceph txt unknown /proc/56498/task/57894/exe (readlink: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57894 tp_osd_tp ceph NOFD /proc/56498/task/57894/fd (opendir: Permission denied) 2024-04-22T06:40:17.737 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57895 tp_osd_tp ceph cwd unknown /proc/56498/task/57895/cwd (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57895 tp_osd_tp ceph rtd unknown /proc/56498/task/57895/root (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57895 tp_osd_tp ceph txt unknown /proc/56498/task/57895/exe (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57895 tp_osd_tp ceph NOFD /proc/56498/task/57895/fd (opendir: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57896 tp_osd_tp ceph cwd unknown /proc/56498/task/57896/cwd (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57896 tp_osd_tp ceph rtd unknown /proc/56498/task/57896/root (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57896 tp_osd_tp ceph txt unknown /proc/56498/task/57896/exe (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57896 tp_osd_tp ceph NOFD /proc/56498/task/57896/fd (opendir: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57897 tp_osd_tp ceph cwd unknown /proc/56498/task/57897/cwd (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57897 tp_osd_tp ceph rtd unknown /proc/56498/task/57897/root (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57897 tp_osd_tp ceph txt unknown /proc/56498/task/57897/exe (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57897 tp_osd_tp ceph NOFD /proc/56498/task/57897/fd (opendir: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57898 tp_osd_tp ceph cwd unknown /proc/56498/task/57898/cwd (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57898 tp_osd_tp ceph rtd unknown /proc/56498/task/57898/root (readlink: Permission denied) 2024-04-22T06:40:17.738 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57898 tp_osd_tp ceph txt unknown /proc/56498/task/57898/exe (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57898 tp_osd_tp ceph NOFD /proc/56498/task/57898/fd (opendir: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57899 tp_osd_tp ceph cwd unknown /proc/56498/task/57899/cwd (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57899 tp_osd_tp ceph rtd unknown /proc/56498/task/57899/root (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57899 tp_osd_tp ceph txt unknown /proc/56498/task/57899/exe (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57899 tp_osd_tp ceph NOFD /proc/56498/task/57899/fd (opendir: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57900 tp_osd_tp ceph cwd unknown /proc/56498/task/57900/cwd (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57900 tp_osd_tp ceph rtd unknown /proc/56498/task/57900/root (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57900 tp_osd_tp ceph txt unknown /proc/56498/task/57900/exe (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57900 tp_osd_tp ceph NOFD /proc/56498/task/57900/fd (opendir: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57901 tp_osd_tp ceph cwd unknown /proc/56498/task/57901/cwd (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57901 tp_osd_tp ceph rtd unknown /proc/56498/task/57901/root (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57901 tp_osd_tp ceph txt unknown /proc/56498/task/57901/exe (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57901 tp_osd_tp ceph NOFD /proc/56498/task/57901/fd (opendir: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57902 tp_osd_tp ceph cwd unknown /proc/56498/task/57902/cwd (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57902 tp_osd_tp ceph rtd unknown /proc/56498/task/57902/root (readlink: Permission denied) 2024-04-22T06:40:17.739 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57902 tp_osd_tp ceph txt unknown /proc/56498/task/57902/exe (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57902 tp_osd_tp ceph NOFD /proc/56498/task/57902/fd (opendir: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57903 tp_osd_tp ceph cwd unknown /proc/56498/task/57903/cwd (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57903 tp_osd_tp ceph rtd unknown /proc/56498/task/57903/root (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57903 tp_osd_tp ceph txt unknown /proc/56498/task/57903/exe (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57903 tp_osd_tp ceph NOFD /proc/56498/task/57903/fd (opendir: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57904 tp_osd_tp ceph cwd unknown /proc/56498/task/57904/cwd (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57904 tp_osd_tp ceph rtd unknown /proc/56498/task/57904/root (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57904 tp_osd_tp ceph txt unknown /proc/56498/task/57904/exe (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57904 tp_osd_tp ceph NOFD /proc/56498/task/57904/fd (opendir: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57905 tp_osd_tp ceph cwd unknown /proc/56498/task/57905/cwd (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57905 tp_osd_tp ceph rtd unknown /proc/56498/task/57905/root (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57905 tp_osd_tp ceph txt unknown /proc/56498/task/57905/exe (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57905 tp_osd_tp ceph NOFD /proc/56498/task/57905/fd (opendir: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57906 tp_osd_tp ceph cwd unknown /proc/56498/task/57906/cwd (readlink: Permission denied) 2024-04-22T06:40:17.740 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57906 tp_osd_tp ceph rtd unknown /proc/56498/task/57906/root (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57906 tp_osd_tp ceph txt unknown /proc/56498/task/57906/exe (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57906 tp_osd_tp ceph NOFD /proc/56498/task/57906/fd (opendir: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57907 osd_srv_h ceph cwd unknown /proc/56498/task/57907/cwd (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57907 osd_srv_h ceph rtd unknown /proc/56498/task/57907/root (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57907 osd_srv_h ceph txt unknown /proc/56498/task/57907/exe (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 56498 57907 osd_srv_h ceph NOFD /proc/56498/task/57907/fd (opendir: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 57926 root cwd unknown /proc/57926/cwd (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 57926 root rtd unknown /proc/57926/root (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 57926 root txt unknown /proc/57926/exe (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 57926 root NOFD /proc/57926/fd (opendir: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 57955 root cwd unknown /proc/57955/cwd (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 57955 root rtd unknown /proc/57955/root (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 57955 root txt unknown /proc/57955/exe (readlink: Permission denied) 2024-04-22T06:40:17.741 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 57955 root NOFD /proc/57955/fd (opendir: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 62902 root cwd unknown /proc/62902/cwd (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 62902 root rtd unknown /proc/62902/root (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 62902 root txt unknown /proc/62902/exe (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/7 62902 root NOFD /proc/62902/fd (opendir: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 63514 root cwd unknown /proc/63514/cwd (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 63514 root rtd unknown /proc/63514/root (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 63514 root txt unknown /proc/63514/exe (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 63514 root NOFD /proc/63514/fd (opendir: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 63516 root cwd unknown /proc/63516/cwd (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 63516 root rtd unknown /proc/63516/root (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 63516 root txt unknown /proc/63516/exe (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 63516 root NOFD /proc/63516/fd (opendir: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 ceph cwd unknown /proc/63518/cwd (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 ceph rtd unknown /proc/63518/root (readlink: Permission denied) 2024-04-22T06:40:17.742 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 ceph txt unknown /proc/63518/exe (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 ceph NOFD /proc/63518/fd (opendir: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63519 log ceph cwd unknown /proc/63518/task/63519/cwd (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63519 log ceph rtd unknown /proc/63518/task/63519/root (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63519 log ceph txt unknown /proc/63518/task/63519/exe (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63519 log ceph NOFD /proc/63518/task/63519/fd (opendir: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63521 msgr-work ceph cwd unknown /proc/63518/task/63521/cwd (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63521 msgr-work ceph rtd unknown /proc/63518/task/63521/root (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63521 msgr-work ceph txt unknown /proc/63518/task/63521/exe (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63521 msgr-work ceph NOFD /proc/63518/task/63521/fd (opendir: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63522 msgr-work ceph cwd unknown /proc/63518/task/63522/cwd (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63522 msgr-work ceph rtd unknown /proc/63518/task/63522/root (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63522 msgr-work ceph txt unknown /proc/63518/task/63522/exe (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63522 msgr-work ceph NOFD /proc/63518/task/63522/fd (opendir: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63523 msgr-work ceph cwd unknown /proc/63518/task/63523/cwd (readlink: Permission denied) 2024-04-22T06:40:17.743 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63523 msgr-work ceph rtd unknown /proc/63518/task/63523/root (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63523 msgr-work ceph txt unknown /proc/63518/task/63523/exe (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63523 msgr-work ceph NOFD /proc/63518/task/63523/fd (opendir: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63527 service ceph cwd unknown /proc/63518/task/63527/cwd (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63527 service ceph rtd unknown /proc/63518/task/63527/root (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63527 service ceph txt unknown /proc/63518/task/63527/exe (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63527 service ceph NOFD /proc/63518/task/63527/fd (opendir: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63528 admin_soc ceph cwd unknown /proc/63518/task/63528/cwd (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63528 admin_soc ceph rtd unknown /proc/63518/task/63528/root (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63528 admin_soc ceph txt unknown /proc/63518/task/63528/exe (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63528 admin_soc ceph NOFD /proc/63518/task/63528/fd (opendir: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63670 signal_ha ceph cwd unknown /proc/63518/task/63670/cwd (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63670 signal_ha ceph rtd unknown /proc/63518/task/63670/root (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63670 signal_ha ceph txt unknown /proc/63518/task/63670/exe (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63670 signal_ha ceph NOFD /proc/63518/task/63670/fd (opendir: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63671 io_contex ceph cwd unknown /proc/63518/task/63671/cwd (readlink: Permission denied) 2024-04-22T06:40:17.744 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63671 io_contex ceph rtd unknown /proc/63518/task/63671/root (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63671 io_contex ceph txt unknown /proc/63518/task/63671/exe (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63671 io_contex ceph NOFD /proc/63518/task/63671/fd (opendir: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63672 io_contex ceph cwd unknown /proc/63518/task/63672/cwd (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63672 io_contex ceph rtd unknown /proc/63518/task/63672/root (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63672 io_contex ceph txt unknown /proc/63518/task/63672/exe (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63672 io_contex ceph NOFD /proc/63518/task/63672/fd (opendir: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63783 OpHistory ceph cwd unknown /proc/63518/task/63783/cwd (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63783 OpHistory ceph rtd unknown /proc/63518/task/63783/root (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63783 OpHistory ceph txt unknown /proc/63518/task/63783/exe (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63783 OpHistory ceph NOFD /proc/63518/task/63783/fd (opendir: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63784 ceph_time ceph cwd unknown /proc/63518/task/63784/cwd (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63784 ceph_time ceph rtd unknown /proc/63518/task/63784/root (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63784 ceph_time ceph txt unknown /proc/63518/task/63784/exe (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63784 ceph_time ceph NOFD /proc/63518/task/63784/fd (opendir: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63819 ceph-osd ceph cwd unknown /proc/63518/task/63819/cwd (readlink: Permission denied) 2024-04-22T06:40:17.745 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63819 ceph-osd ceph rtd unknown /proc/63518/task/63819/root (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63819 ceph-osd ceph txt unknown /proc/63518/task/63819/exe (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63819 ceph-osd ceph NOFD /proc/63518/task/63819/fd (opendir: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63839 ceph-osd ceph cwd unknown /proc/63518/task/63839/cwd (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63839 ceph-osd ceph rtd unknown /proc/63518/task/63839/root (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63839 ceph-osd ceph txt unknown /proc/63518/task/63839/exe (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63839 ceph-osd ceph NOFD /proc/63518/task/63839/fd (opendir: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63842 ceph-osd ceph cwd unknown /proc/63518/task/63842/cwd (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63842 ceph-osd ceph rtd unknown /proc/63518/task/63842/root (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63842 ceph-osd ceph txt unknown /proc/63518/task/63842/exe (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63842 ceph-osd ceph NOFD /proc/63518/task/63842/fd (opendir: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63868 ceph-osd ceph cwd unknown /proc/63518/task/63868/cwd (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63868 ceph-osd ceph rtd unknown /proc/63518/task/63868/root (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63868 ceph-osd ceph txt unknown /proc/63518/task/63868/exe (readlink: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63868 ceph-osd ceph NOFD /proc/63518/task/63868/fd (opendir: Permission denied) 2024-04-22T06:40:17.746 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63876 ceph-osd ceph cwd unknown /proc/63518/task/63876/cwd (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63876 ceph-osd ceph rtd unknown /proc/63518/task/63876/root (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63876 ceph-osd ceph txt unknown /proc/63518/task/63876/exe (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63876 ceph-osd ceph NOFD /proc/63518/task/63876/fd (opendir: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63886 ceph-osd ceph cwd unknown /proc/63518/task/63886/cwd (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63886 ceph-osd ceph rtd unknown /proc/63518/task/63886/root (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63886 ceph-osd ceph txt unknown /proc/63518/task/63886/exe (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63886 ceph-osd ceph NOFD /proc/63518/task/63886/fd (opendir: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63944 ceph-osd ceph cwd unknown /proc/63518/task/63944/cwd (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63944 ceph-osd ceph rtd unknown /proc/63518/task/63944/root (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63944 ceph-osd ceph txt unknown /proc/63518/task/63944/exe (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63944 ceph-osd ceph NOFD /proc/63518/task/63944/fd (opendir: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63969 ceph-osd ceph cwd unknown /proc/63518/task/63969/cwd (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63969 ceph-osd ceph rtd unknown /proc/63518/task/63969/root (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63969 ceph-osd ceph txt unknown /proc/63518/task/63969/exe (readlink: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63969 ceph-osd ceph NOFD /proc/63518/task/63969/fd (opendir: Permission denied) 2024-04-22T06:40:17.747 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63970 ceph-osd ceph cwd unknown /proc/63518/task/63970/cwd (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63970 ceph-osd ceph rtd unknown /proc/63518/task/63970/root (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63970 ceph-osd ceph txt unknown /proc/63518/task/63970/exe (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63970 ceph-osd ceph NOFD /proc/63518/task/63970/fd (opendir: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63971 safe_time ceph cwd unknown /proc/63518/task/63971/cwd (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63971 safe_time ceph rtd unknown /proc/63518/task/63971/root (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63971 safe_time ceph txt unknown /proc/63518/task/63971/exe (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63971 safe_time ceph NOFD /proc/63518/task/63971/fd (opendir: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63972 safe_time ceph cwd unknown /proc/63518/task/63972/cwd (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63972 safe_time ceph rtd unknown /proc/63518/task/63972/root (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63972 safe_time ceph txt unknown /proc/63518/task/63972/exe (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63972 safe_time ceph NOFD /proc/63518/task/63972/fd (opendir: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63973 safe_time ceph cwd unknown /proc/63518/task/63973/cwd (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63973 safe_time ceph rtd unknown /proc/63518/task/63973/root (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63973 safe_time ceph txt unknown /proc/63518/task/63973/exe (readlink: Permission denied) 2024-04-22T06:40:17.748 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63973 safe_time ceph NOFD /proc/63518/task/63973/fd (opendir: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63974 safe_time ceph cwd unknown /proc/63518/task/63974/cwd (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63974 safe_time ceph rtd unknown /proc/63518/task/63974/root (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63974 safe_time ceph txt unknown /proc/63518/task/63974/exe (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63974 safe_time ceph NOFD /proc/63518/task/63974/fd (opendir: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63975 fn_anonym ceph cwd unknown /proc/63518/task/63975/cwd (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63975 fn_anonym ceph rtd unknown /proc/63518/task/63975/root (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63975 fn_anonym ceph txt unknown /proc/63518/task/63975/exe (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63975 fn_anonym ceph NOFD /proc/63518/task/63975/fd (opendir: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63981 rocksdb:l ceph cwd unknown /proc/63518/task/63981/cwd (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63981 rocksdb:l ceph rtd unknown /proc/63518/task/63981/root (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63981 rocksdb:l ceph txt unknown /proc/63518/task/63981/exe (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63981 rocksdb:l ceph NOFD /proc/63518/task/63981/fd (opendir: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63982 rocksdb:l ceph cwd unknown /proc/63518/task/63982/cwd (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63982 rocksdb:l ceph rtd unknown /proc/63518/task/63982/root (readlink: Permission denied) 2024-04-22T06:40:17.749 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63982 rocksdb:l ceph txt unknown /proc/63518/task/63982/exe (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63982 rocksdb:l ceph NOFD /proc/63518/task/63982/fd (opendir: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63983 rocksdb:l ceph cwd unknown /proc/63518/task/63983/cwd (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63983 rocksdb:l ceph rtd unknown /proc/63518/task/63983/root (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63983 rocksdb:l ceph txt unknown /proc/63518/task/63983/exe (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63983 rocksdb:l ceph NOFD /proc/63518/task/63983/fd (opendir: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63984 rocksdb:h ceph cwd unknown /proc/63518/task/63984/cwd (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63984 rocksdb:h ceph rtd unknown /proc/63518/task/63984/root (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63984 rocksdb:h ceph txt unknown /proc/63518/task/63984/exe (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 63984 rocksdb:h ceph NOFD /proc/63518/task/63984/fd (opendir: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64415 bstore_ai ceph cwd unknown /proc/63518/task/64415/cwd (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64415 bstore_ai ceph rtd unknown /proc/63518/task/64415/root (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64415 bstore_ai ceph txt unknown /proc/63518/task/64415/exe (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64415 bstore_ai ceph NOFD /proc/63518/task/64415/fd (opendir: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64620 bstore_ai ceph cwd unknown /proc/63518/task/64620/cwd (readlink: Permission denied) 2024-04-22T06:40:17.750 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64620 bstore_ai ceph rtd unknown /proc/63518/task/64620/root (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64620 bstore_ai ceph txt unknown /proc/63518/task/64620/exe (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64620 bstore_ai ceph NOFD /proc/63518/task/64620/fd (opendir: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64802 ceph-osd ceph cwd unknown /proc/63518/task/64802/cwd (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64802 ceph-osd ceph rtd unknown /proc/63518/task/64802/root (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64802 ceph-osd ceph txt unknown /proc/63518/task/64802/exe (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64802 ceph-osd ceph NOFD /proc/63518/task/64802/fd (opendir: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64803 cfin ceph cwd unknown /proc/63518/task/64803/cwd (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64803 cfin ceph rtd unknown /proc/63518/task/64803/root (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64803 cfin ceph txt unknown /proc/63518/task/64803/exe (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64803 cfin ceph NOFD /proc/63518/task/64803/fd (opendir: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64804 bstore_kv ceph cwd unknown /proc/63518/task/64804/cwd (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64804 bstore_kv ceph rtd unknown /proc/63518/task/64804/root (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64804 bstore_kv ceph txt unknown /proc/63518/task/64804/exe (readlink: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64804 bstore_kv ceph NOFD /proc/63518/task/64804/fd (opendir: Permission denied) 2024-04-22T06:40:17.751 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64805 bstore_kv ceph cwd unknown /proc/63518/task/64805/cwd (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64805 bstore_kv ceph rtd unknown /proc/63518/task/64805/root (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64805 bstore_kv ceph txt unknown /proc/63518/task/64805/exe (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64805 bstore_kv ceph NOFD /proc/63518/task/64805/fd (opendir: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64806 bstore_me ceph cwd unknown /proc/63518/task/64806/cwd (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64806 bstore_me ceph rtd unknown /proc/63518/task/64806/root (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64806 bstore_me ceph txt unknown /proc/63518/task/64806/exe (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64806 bstore_me ceph NOFD /proc/63518/task/64806/fd (opendir: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64807 ms_dispat ceph cwd unknown /proc/63518/task/64807/cwd (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64807 ms_dispat ceph rtd unknown /proc/63518/task/64807/root (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64807 ms_dispat ceph txt unknown /proc/63518/task/64807/exe (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64807 ms_dispat ceph NOFD /proc/63518/task/64807/fd (opendir: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64808 ms_local ceph cwd unknown /proc/63518/task/64808/cwd (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64808 ms_local ceph rtd unknown /proc/63518/task/64808/root (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64808 ms_local ceph txt unknown /proc/63518/task/64808/exe (readlink: Permission denied) 2024-04-22T06:40:17.752 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64808 ms_local ceph NOFD /proc/63518/task/64808/fd (opendir: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64809 safe_time ceph cwd unknown /proc/63518/task/64809/cwd (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64809 safe_time ceph rtd unknown /proc/63518/task/64809/root (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64809 safe_time ceph txt unknown /proc/63518/task/64809/exe (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64809 safe_time ceph NOFD /proc/63518/task/64809/fd (opendir: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64810 safe_time ceph cwd unknown /proc/63518/task/64810/cwd (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64810 safe_time ceph rtd unknown /proc/63518/task/64810/root (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64810 safe_time ceph txt unknown /proc/63518/task/64810/exe (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64810 safe_time ceph NOFD /proc/63518/task/64810/fd (opendir: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64811 ms_dispat ceph cwd unknown /proc/63518/task/64811/cwd (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64811 ms_dispat ceph rtd unknown /proc/63518/task/64811/root (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64811 ms_dispat ceph txt unknown /proc/63518/task/64811/exe (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64811 ms_dispat ceph NOFD /proc/63518/task/64811/fd (opendir: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64812 ms_local ceph cwd unknown /proc/63518/task/64812/cwd (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64812 ms_local ceph rtd unknown /proc/63518/task/64812/root (readlink: Permission denied) 2024-04-22T06:40:17.753 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64812 ms_local ceph txt unknown /proc/63518/task/64812/exe (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64812 ms_local ceph NOFD /proc/63518/task/64812/fd (opendir: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64813 ms_dispat ceph cwd unknown /proc/63518/task/64813/cwd (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64813 ms_dispat ceph rtd unknown /proc/63518/task/64813/root (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64813 ms_dispat ceph txt unknown /proc/63518/task/64813/exe (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64813 ms_dispat ceph NOFD /proc/63518/task/64813/fd (opendir: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64814 ms_local ceph cwd unknown /proc/63518/task/64814/cwd (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64814 ms_local ceph rtd unknown /proc/63518/task/64814/root (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64814 ms_local ceph txt unknown /proc/63518/task/64814/exe (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64814 ms_local ceph NOFD /proc/63518/task/64814/fd (opendir: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64815 ms_dispat ceph cwd unknown /proc/63518/task/64815/cwd (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64815 ms_dispat ceph rtd unknown /proc/63518/task/64815/root (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64815 ms_dispat ceph txt unknown /proc/63518/task/64815/exe (readlink: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64815 ms_dispat ceph NOFD /proc/63518/task/64815/fd (opendir: Permission denied) 2024-04-22T06:40:17.754 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64816 ms_local ceph cwd unknown /proc/63518/task/64816/cwd (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64816 ms_local ceph rtd unknown /proc/63518/task/64816/root (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64816 ms_local ceph txt unknown /proc/63518/task/64816/exe (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64816 ms_local ceph NOFD /proc/63518/task/64816/fd (opendir: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64817 ms_dispat ceph cwd unknown /proc/63518/task/64817/cwd (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64817 ms_dispat ceph rtd unknown /proc/63518/task/64817/root (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64817 ms_dispat ceph txt unknown /proc/63518/task/64817/exe (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64817 ms_dispat ceph NOFD /proc/63518/task/64817/fd (opendir: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64818 ms_local ceph cwd unknown /proc/63518/task/64818/cwd (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64818 ms_local ceph rtd unknown /proc/63518/task/64818/root (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64818 ms_local ceph txt unknown /proc/63518/task/64818/exe (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64818 ms_local ceph NOFD /proc/63518/task/64818/fd (opendir: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64819 ms_dispat ceph cwd unknown /proc/63518/task/64819/cwd (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64819 ms_dispat ceph rtd unknown /proc/63518/task/64819/root (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64819 ms_dispat ceph txt unknown /proc/63518/task/64819/exe (readlink: Permission denied) 2024-04-22T06:40:17.755 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64819 ms_dispat ceph NOFD /proc/63518/task/64819/fd (opendir: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64820 ms_local ceph cwd unknown /proc/63518/task/64820/cwd (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64820 ms_local ceph rtd unknown /proc/63518/task/64820/root (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64820 ms_local ceph txt unknown /proc/63518/task/64820/exe (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64820 ms_local ceph NOFD /proc/63518/task/64820/fd (opendir: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64821 ms_dispat ceph cwd unknown /proc/63518/task/64821/cwd (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64821 ms_dispat ceph rtd unknown /proc/63518/task/64821/root (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64821 ms_dispat ceph txt unknown /proc/63518/task/64821/exe (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64821 ms_dispat ceph NOFD /proc/63518/task/64821/fd (opendir: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64822 ms_local ceph cwd unknown /proc/63518/task/64822/cwd (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64822 ms_local ceph rtd unknown /proc/63518/task/64822/root (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64822 ms_local ceph txt unknown /proc/63518/task/64822/exe (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64822 ms_local ceph NOFD /proc/63518/task/64822/fd (opendir: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64823 fn_anonym ceph cwd unknown /proc/63518/task/64823/cwd (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64823 fn_anonym ceph rtd unknown /proc/63518/task/64823/root (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64823 fn_anonym ceph txt unknown /proc/63518/task/64823/exe (readlink: Permission denied) 2024-04-22T06:40:17.756 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64823 fn_anonym ceph NOFD /proc/63518/task/64823/fd (opendir: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64824 finisher ceph cwd unknown /proc/63518/task/64824/cwd (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64824 finisher ceph rtd unknown /proc/63518/task/64824/root (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64824 finisher ceph txt unknown /proc/63518/task/64824/exe (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64824 finisher ceph NOFD /proc/63518/task/64824/fd (opendir: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64825 safe_time ceph cwd unknown /proc/63518/task/64825/cwd (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64825 safe_time ceph rtd unknown /proc/63518/task/64825/root (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64825 safe_time ceph txt unknown /proc/63518/task/64825/exe (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64825 safe_time ceph NOFD /proc/63518/task/64825/fd (opendir: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64826 safe_time ceph cwd unknown /proc/63518/task/64826/cwd (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64826 safe_time ceph rtd unknown /proc/63518/task/64826/root (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64826 safe_time ceph txt unknown /proc/63518/task/64826/exe (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64826 safe_time ceph NOFD /proc/63518/task/64826/fd (opendir: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64827 ceph-osd ceph cwd unknown /proc/63518/task/64827/cwd (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64827 ceph-osd ceph rtd unknown /proc/63518/task/64827/root (readlink: Permission denied) 2024-04-22T06:40:17.757 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64827 ceph-osd ceph txt unknown /proc/63518/task/64827/exe (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64827 ceph-osd ceph NOFD /proc/63518/task/64827/fd (opendir: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64828 osd_srv_a ceph cwd unknown /proc/63518/task/64828/cwd (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64828 osd_srv_a ceph rtd unknown /proc/63518/task/64828/root (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64828 osd_srv_a ceph txt unknown /proc/63518/task/64828/exe (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64828 osd_srv_a ceph NOFD /proc/63518/task/64828/fd (opendir: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64829 tp_osd_tp ceph cwd unknown /proc/63518/task/64829/cwd (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64829 tp_osd_tp ceph rtd unknown /proc/63518/task/64829/root (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64829 tp_osd_tp ceph txt unknown /proc/63518/task/64829/exe (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64829 tp_osd_tp ceph NOFD /proc/63518/task/64829/fd (opendir: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64830 tp_osd_tp ceph cwd unknown /proc/63518/task/64830/cwd (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64830 tp_osd_tp ceph rtd unknown /proc/63518/task/64830/root (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64830 tp_osd_tp ceph txt unknown /proc/63518/task/64830/exe (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64830 tp_osd_tp ceph NOFD /proc/63518/task/64830/fd (opendir: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64831 tp_osd_tp ceph cwd unknown /proc/63518/task/64831/cwd (readlink: Permission denied) 2024-04-22T06:40:17.758 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64831 tp_osd_tp ceph rtd unknown /proc/63518/task/64831/root (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64831 tp_osd_tp ceph txt unknown /proc/63518/task/64831/exe (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64831 tp_osd_tp ceph NOFD /proc/63518/task/64831/fd (opendir: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64832 tp_osd_tp ceph cwd unknown /proc/63518/task/64832/cwd (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64832 tp_osd_tp ceph rtd unknown /proc/63518/task/64832/root (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64832 tp_osd_tp ceph txt unknown /proc/63518/task/64832/exe (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64832 tp_osd_tp ceph NOFD /proc/63518/task/64832/fd (opendir: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64833 tp_osd_tp ceph cwd unknown /proc/63518/task/64833/cwd (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64833 tp_osd_tp ceph rtd unknown /proc/63518/task/64833/root (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64833 tp_osd_tp ceph txt unknown /proc/63518/task/64833/exe (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64833 tp_osd_tp ceph NOFD /proc/63518/task/64833/fd (opendir: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64834 tp_osd_tp ceph cwd unknown /proc/63518/task/64834/cwd (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64834 tp_osd_tp ceph rtd unknown /proc/63518/task/64834/root (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64834 tp_osd_tp ceph txt unknown /proc/63518/task/64834/exe (readlink: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64834 tp_osd_tp ceph NOFD /proc/63518/task/64834/fd (opendir: Permission denied) 2024-04-22T06:40:17.759 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64835 tp_osd_tp ceph cwd unknown /proc/63518/task/64835/cwd (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64835 tp_osd_tp ceph rtd unknown /proc/63518/task/64835/root (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64835 tp_osd_tp ceph txt unknown /proc/63518/task/64835/exe (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64835 tp_osd_tp ceph NOFD /proc/63518/task/64835/fd (opendir: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64836 tp_osd_tp ceph cwd unknown /proc/63518/task/64836/cwd (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64836 tp_osd_tp ceph rtd unknown /proc/63518/task/64836/root (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64836 tp_osd_tp ceph txt unknown /proc/63518/task/64836/exe (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64836 tp_osd_tp ceph NOFD /proc/63518/task/64836/fd (opendir: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64837 tp_osd_tp ceph cwd unknown /proc/63518/task/64837/cwd (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64837 tp_osd_tp ceph rtd unknown /proc/63518/task/64837/root (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64837 tp_osd_tp ceph txt unknown /proc/63518/task/64837/exe (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64837 tp_osd_tp ceph NOFD /proc/63518/task/64837/fd (opendir: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64838 tp_osd_tp ceph cwd unknown /proc/63518/task/64838/cwd (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64838 tp_osd_tp ceph rtd unknown /proc/63518/task/64838/root (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64838 tp_osd_tp ceph txt unknown /proc/63518/task/64838/exe (readlink: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64838 tp_osd_tp ceph NOFD /proc/63518/task/64838/fd (opendir: Permission denied) 2024-04-22T06:40:17.760 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64839 tp_osd_tp ceph cwd unknown /proc/63518/task/64839/cwd (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64839 tp_osd_tp ceph rtd unknown /proc/63518/task/64839/root (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64839 tp_osd_tp ceph txt unknown /proc/63518/task/64839/exe (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64839 tp_osd_tp ceph NOFD /proc/63518/task/64839/fd (opendir: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64840 tp_osd_tp ceph cwd unknown /proc/63518/task/64840/cwd (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64840 tp_osd_tp ceph rtd unknown /proc/63518/task/64840/root (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64840 tp_osd_tp ceph txt unknown /proc/63518/task/64840/exe (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64840 tp_osd_tp ceph NOFD /proc/63518/task/64840/fd (opendir: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64841 tp_osd_tp ceph cwd unknown /proc/63518/task/64841/cwd (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64841 tp_osd_tp ceph rtd unknown /proc/63518/task/64841/root (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64841 tp_osd_tp ceph txt unknown /proc/63518/task/64841/exe (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64841 tp_osd_tp ceph NOFD /proc/63518/task/64841/fd (opendir: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64842 tp_osd_tp ceph cwd unknown /proc/63518/task/64842/cwd (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64842 tp_osd_tp ceph rtd unknown /proc/63518/task/64842/root (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64842 tp_osd_tp ceph txt unknown /proc/63518/task/64842/exe (readlink: Permission denied) 2024-04-22T06:40:17.761 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64842 tp_osd_tp ceph NOFD /proc/63518/task/64842/fd (opendir: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64843 tp_osd_tp ceph cwd unknown /proc/63518/task/64843/cwd (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64843 tp_osd_tp ceph rtd unknown /proc/63518/task/64843/root (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64843 tp_osd_tp ceph txt unknown /proc/63518/task/64843/exe (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64843 tp_osd_tp ceph NOFD /proc/63518/task/64843/fd (opendir: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64844 tp_osd_tp ceph cwd unknown /proc/63518/task/64844/cwd (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64844 tp_osd_tp ceph rtd unknown /proc/63518/task/64844/root (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64844 tp_osd_tp ceph txt unknown /proc/63518/task/64844/exe (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64844 tp_osd_tp ceph NOFD /proc/63518/task/64844/fd (opendir: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64845 osd_srv_h ceph cwd unknown /proc/63518/task/64845/cwd (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64845 osd_srv_h ceph rtd unknown /proc/63518/task/64845/root (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64845 osd_srv_h ceph txt unknown /proc/63518/task/64845/exe (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-osd 63518 64845 osd_srv_h ceph NOFD /proc/63518/task/64845/fd (opendir: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 63846 root cwd unknown /proc/63846/cwd (readlink: Permission denied) 2024-04-22T06:40:17.762 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 63846 root rtd unknown /proc/63846/root (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 63846 root txt unknown /proc/63846/exe (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/u 63846 root NOFD /proc/63846/fd (opendir: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 64847 root cwd unknown /proc/64847/cwd (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 64847 root rtd unknown /proc/64847/root (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 64847 root txt unknown /proc/64847/exe (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 64847 root NOFD /proc/64847/fd (opendir: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 64864 root cwd unknown /proc/64864/cwd (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 64864 root rtd unknown /proc/64864/root (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 64864 root txt unknown /proc/64864/exe (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 64864 root NOFD /proc/64864/fd (opendir: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 65031 root cwd unknown /proc/65031/cwd (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 65031 root rtd unknown /proc/65031/root (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 65031 root txt unknown /proc/65031/exe (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 65031 root NOFD /proc/65031/fd (opendir: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68324 root cwd unknown /proc/68324/cwd (readlink: Permission denied) 2024-04-22T06:40:17.763 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68324 root rtd unknown /proc/68324/root (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68324 root txt unknown /proc/68324/exe (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68324 root NOFD /proc/68324/fd (opendir: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68341 root cwd unknown /proc/68341/cwd (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68341 root rtd unknown /proc/68341/root (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68341 root txt unknown /proc/68341/exe (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68341 root NOFD /proc/68341/fd (opendir: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68342 root cwd unknown /proc/68342/cwd (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68342 root rtd unknown /proc/68342/root (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68342 root txt unknown /proc/68342/exe (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68342 root NOFD /proc/68342/fd (opendir: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68359 root cwd unknown /proc/68359/cwd (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68359 root rtd unknown /proc/68359/root (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68359 root txt unknown /proc/68359/exe (readlink: Permission denied) 2024-04-22T06:40:17.764 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68359 root NOFD /proc/68359/fd (opendir: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68360 root cwd unknown /proc/68360/cwd (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68360 root rtd unknown /proc/68360/root (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68360 root txt unknown /proc/68360/exe (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68360 root NOFD /proc/68360/fd (opendir: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68377 root cwd unknown /proc/68377/cwd (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68377 root rtd unknown /proc/68377/root (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68377 root txt unknown /proc/68377/exe (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68377 root NOFD /proc/68377/fd (opendir: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68378 root cwd unknown /proc/68378/cwd (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68378 root rtd unknown /proc/68378/root (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68378 root txt unknown /proc/68378/exe (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 68378 root NOFD /proc/68378/fd (opendir: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68395 root cwd unknown /proc/68395/cwd (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68395 root rtd unknown /proc/68395/root (readlink: Permission denied) 2024-04-22T06:40:17.765 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68395 root txt unknown /proc/68395/exe (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:journalct 68395 root NOFD /proc/68395/fd (opendir: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 68879 root cwd unknown /proc/68879/cwd (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 68879 root rtd unknown /proc/68879/root (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 68879 root txt unknown /proc/68879/exe (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 68879 root NOFD /proc/68879/fd (opendir: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 68881 root cwd unknown /proc/68881/cwd (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 68881 root rtd unknown /proc/68881/root (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 68881 root txt unknown /proc/68881/exe (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 68881 root NOFD /proc/68881/fd (opendir: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 ceph cwd unknown /proc/68883/cwd (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 ceph rtd unknown /proc/68883/root (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 ceph txt unknown /proc/68883/exe (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 ceph NOFD /proc/68883/fd (opendir: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68884 log ceph cwd unknown /proc/68883/task/68884/cwd (readlink: Permission denied) 2024-04-22T06:40:17.766 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68884 log ceph rtd unknown /proc/68883/task/68884/root (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68884 log ceph txt unknown /proc/68883/task/68884/exe (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68884 log ceph NOFD /proc/68883/task/68884/fd (opendir: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68886 msgr-work ceph cwd unknown /proc/68883/task/68886/cwd (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68886 msgr-work ceph rtd unknown /proc/68883/task/68886/root (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68886 msgr-work ceph txt unknown /proc/68883/task/68886/exe (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68886 msgr-work ceph NOFD /proc/68883/task/68886/fd (opendir: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68887 msgr-work ceph cwd unknown /proc/68883/task/68887/cwd (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68887 msgr-work ceph rtd unknown /proc/68883/task/68887/root (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68887 msgr-work ceph txt unknown /proc/68883/task/68887/exe (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68887 msgr-work ceph NOFD /proc/68883/task/68887/fd (opendir: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68888 msgr-work ceph cwd unknown /proc/68883/task/68888/cwd (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68888 msgr-work ceph rtd unknown /proc/68883/task/68888/root (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68888 msgr-work ceph txt unknown /proc/68883/task/68888/exe (readlink: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68888 msgr-work ceph NOFD /proc/68883/task/68888/fd (opendir: Permission denied) 2024-04-22T06:40:17.767 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68892 service ceph cwd unknown /proc/68883/task/68892/cwd (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68892 service ceph rtd unknown /proc/68883/task/68892/root (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68892 service ceph txt unknown /proc/68883/task/68892/exe (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68892 service ceph NOFD /proc/68883/task/68892/fd (opendir: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68893 admin_soc ceph cwd unknown /proc/68883/task/68893/cwd (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68893 admin_soc ceph rtd unknown /proc/68883/task/68893/root (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68893 admin_soc ceph txt unknown /proc/68883/task/68893/exe (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68893 admin_soc ceph NOFD /proc/68883/task/68893/fd (opendir: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68894 signal_ha ceph cwd unknown /proc/68883/task/68894/cwd (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68894 signal_ha ceph rtd unknown /proc/68883/task/68894/root (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68894 signal_ha ceph txt unknown /proc/68883/task/68894/exe (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68894 signal_ha ceph NOFD /proc/68883/task/68894/fd (opendir: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68895 io_contex ceph cwd unknown /proc/68883/task/68895/cwd (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68895 io_contex ceph rtd unknown /proc/68883/task/68895/root (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68895 io_contex ceph txt unknown /proc/68883/task/68895/exe (readlink: Permission denied) 2024-04-22T06:40:17.768 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68895 io_contex ceph NOFD /proc/68883/task/68895/fd (opendir: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68896 io_contex ceph cwd unknown /proc/68883/task/68896/cwd (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68896 io_contex ceph rtd unknown /proc/68883/task/68896/root (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68896 io_contex ceph txt unknown /proc/68883/task/68896/exe (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68896 io_contex ceph NOFD /proc/68883/task/68896/fd (opendir: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68897 ms_dispat ceph cwd unknown /proc/68883/task/68897/cwd (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68897 ms_dispat ceph rtd unknown /proc/68883/task/68897/root (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68897 ms_dispat ceph txt unknown /proc/68883/task/68897/exe (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68897 ms_dispat ceph NOFD /proc/68883/task/68897/fd (opendir: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68898 ms_local ceph cwd unknown /proc/68883/task/68898/cwd (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68898 ms_local ceph rtd unknown /proc/68883/task/68898/root (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68898 ms_local ceph txt unknown /proc/68883/task/68898/exe (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68898 ms_local ceph NOFD /proc/68883/task/68898/fd (opendir: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68899 safe_time ceph cwd unknown /proc/68883/task/68899/cwd (readlink: Permission denied) 2024-04-22T06:40:17.769 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68899 safe_time ceph rtd unknown /proc/68883/task/68899/root (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68899 safe_time ceph txt unknown /proc/68883/task/68899/exe (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68899 safe_time ceph NOFD /proc/68883/task/68899/fd (opendir: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68900 safe_time ceph cwd unknown /proc/68883/task/68900/cwd (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68900 safe_time ceph rtd unknown /proc/68883/task/68900/root (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68900 safe_time ceph txt unknown /proc/68883/task/68900/exe (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68900 safe_time ceph NOFD /proc/68883/task/68900/fd (opendir: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68901 ceph-mds ceph cwd unknown /proc/68883/task/68901/cwd (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68901 ceph-mds ceph rtd unknown /proc/68883/task/68901/root (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68901 ceph-mds ceph txt unknown /proc/68883/task/68901/exe (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 68901 ceph-mds ceph NOFD /proc/68883/task/68901/fd (opendir: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 69069 safe_time ceph cwd unknown /proc/68883/task/69069/cwd (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 69069 safe_time ceph rtd unknown /proc/68883/task/69069/root (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 69069 safe_time ceph txt unknown /proc/68883/task/69069/exe (readlink: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 68883 69069 safe_time ceph NOFD /proc/68883/task/69069/fd (opendir: Permission denied) 2024-04-22T06:40:17.770 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 69266 root cwd unknown /proc/69266/cwd (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 69266 root rtd unknown /proc/69266/root (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 69266 root txt unknown /proc/69266/exe (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 69266 root NOFD /proc/69266/fd (opendir: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 69268 root cwd unknown /proc/69268/cwd (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 69268 root rtd unknown /proc/69268/root (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 69268 root txt unknown /proc/69268/exe (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 69268 root NOFD /proc/69268/fd (opendir: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 ceph cwd unknown /proc/69272/cwd (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 ceph rtd unknown /proc/69272/root (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 ceph txt unknown /proc/69272/exe (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 ceph NOFD /proc/69272/fd (opendir: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69273 log ceph cwd unknown /proc/69272/task/69273/cwd (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69273 log ceph rtd unknown /proc/69272/task/69273/root (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69273 log ceph txt unknown /proc/69272/task/69273/exe (readlink: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69273 log ceph NOFD /proc/69272/task/69273/fd (opendir: Permission denied) 2024-04-22T06:40:17.771 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69275 msgr-work ceph cwd unknown /proc/69272/task/69275/cwd (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69275 msgr-work ceph rtd unknown /proc/69272/task/69275/root (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69275 msgr-work ceph txt unknown /proc/69272/task/69275/exe (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69275 msgr-work ceph NOFD /proc/69272/task/69275/fd (opendir: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69276 msgr-work ceph cwd unknown /proc/69272/task/69276/cwd (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69276 msgr-work ceph rtd unknown /proc/69272/task/69276/root (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69276 msgr-work ceph txt unknown /proc/69272/task/69276/exe (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69276 msgr-work ceph NOFD /proc/69272/task/69276/fd (opendir: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69277 msgr-work ceph cwd unknown /proc/69272/task/69277/cwd (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69277 msgr-work ceph rtd unknown /proc/69272/task/69277/root (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69277 msgr-work ceph txt unknown /proc/69272/task/69277/exe (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69277 msgr-work ceph NOFD /proc/69272/task/69277/fd (opendir: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69281 service ceph cwd unknown /proc/69272/task/69281/cwd (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69281 service ceph rtd unknown /proc/69272/task/69281/root (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69281 service ceph txt unknown /proc/69272/task/69281/exe (readlink: Permission denied) 2024-04-22T06:40:17.772 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69281 service ceph NOFD /proc/69272/task/69281/fd (opendir: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69282 admin_soc ceph cwd unknown /proc/69272/task/69282/cwd (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69282 admin_soc ceph rtd unknown /proc/69272/task/69282/root (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69282 admin_soc ceph txt unknown /proc/69272/task/69282/exe (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69282 admin_soc ceph NOFD /proc/69272/task/69282/fd (opendir: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69283 signal_ha ceph cwd unknown /proc/69272/task/69283/cwd (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69283 signal_ha ceph rtd unknown /proc/69272/task/69283/root (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69283 signal_ha ceph txt unknown /proc/69272/task/69283/exe (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69283 signal_ha ceph NOFD /proc/69272/task/69283/fd (opendir: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69284 io_contex ceph cwd unknown /proc/69272/task/69284/cwd (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69284 io_contex ceph rtd unknown /proc/69272/task/69284/root (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69284 io_contex ceph txt unknown /proc/69272/task/69284/exe (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69284 io_contex ceph NOFD /proc/69272/task/69284/fd (opendir: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69285 io_contex ceph cwd unknown /proc/69272/task/69285/cwd (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69285 io_contex ceph rtd unknown /proc/69272/task/69285/root (readlink: Permission denied) 2024-04-22T06:40:17.773 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69285 io_contex ceph txt unknown /proc/69272/task/69285/exe (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69285 io_contex ceph NOFD /proc/69272/task/69285/fd (opendir: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69286 ms_dispat ceph cwd unknown /proc/69272/task/69286/cwd (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69286 ms_dispat ceph rtd unknown /proc/69272/task/69286/root (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69286 ms_dispat ceph txt unknown /proc/69272/task/69286/exe (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69286 ms_dispat ceph NOFD /proc/69272/task/69286/fd (opendir: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69287 ms_local ceph cwd unknown /proc/69272/task/69287/cwd (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69287 ms_local ceph rtd unknown /proc/69272/task/69287/root (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69287 ms_local ceph txt unknown /proc/69272/task/69287/exe (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69287 ms_local ceph NOFD /proc/69272/task/69287/fd (opendir: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69288 safe_time ceph cwd unknown /proc/69272/task/69288/cwd (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69288 safe_time ceph rtd unknown /proc/69272/task/69288/root (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69288 safe_time ceph txt unknown /proc/69272/task/69288/exe (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69288 safe_time ceph NOFD /proc/69272/task/69288/fd (opendir: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69289 safe_time ceph cwd unknown /proc/69272/task/69289/cwd (readlink: Permission denied) 2024-04-22T06:40:17.774 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69289 safe_time ceph rtd unknown /proc/69272/task/69289/root (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69289 safe_time ceph txt unknown /proc/69272/task/69289/exe (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69289 safe_time ceph NOFD /proc/69272/task/69289/fd (opendir: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69290 ceph-mds ceph cwd unknown /proc/69272/task/69290/cwd (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69290 ceph-mds ceph rtd unknown /proc/69272/task/69290/root (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69290 ceph-mds ceph txt unknown /proc/69272/task/69290/exe (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69290 ceph-mds ceph NOFD /proc/69272/task/69290/fd (opendir: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69295 safe_time ceph cwd unknown /proc/69272/task/69295/cwd (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69295 safe_time ceph rtd unknown /proc/69272/task/69295/root (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69295 safe_time ceph txt unknown /proc/69272/task/69295/exe (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 69272 69295 safe_time ceph NOFD /proc/69272/task/69295/fd (opendir: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 72455 root cwd unknown /proc/72455/cwd (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 72455 root rtd unknown /proc/72455/root (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 72455 root txt unknown /proc/72455/exe (readlink: Permission denied) 2024-04-22T06:40:17.775 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/1 72455 root NOFD /proc/72455/fd (opendir: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 73516 root cwd unknown /proc/73516/cwd (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 73516 root rtd unknown /proc/73516/root (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 73516 root txt unknown /proc/73516/exe (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 73516 root NOFD /proc/73516/fd (opendir: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 74768 root cwd unknown /proc/74768/cwd (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 74768 root rtd unknown /proc/74768/root (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 74768 root txt unknown /proc/74768/exe (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 74768 root NOFD /proc/74768/fd (opendir: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 75370 root cwd unknown /proc/75370/cwd (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 75370 root rtd unknown /proc/75370/root (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 75370 root txt unknown /proc/75370/exe (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/0 75370 root NOFD /proc/75370/fd (opendir: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 80123 root cwd unknown /proc/80123/cwd (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 80123 root rtd unknown /proc/80123/root (readlink: Permission denied) 2024-04-22T06:40:17.776 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 80123 root txt unknown /proc/80123/exe (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/6 80123 root NOFD /proc/80123/fd (opendir: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 80239 root cwd unknown /proc/80239/cwd (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 80239 root rtd unknown /proc/80239/root (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 80239 root txt unknown /proc/80239/exe (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/5 80239 root NOFD /proc/80239/fd (opendir: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-msgr 80769 root cwd unknown /proc/80769/cwd (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-msgr 80769 root rtd unknown /proc/80769/root (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-msgr 80769 root txt unknown /proc/80769/exe (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-msgr 80769 root NOFD /proc/80769/fd (opendir: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-watc 80770 root cwd unknown /proc/80770/cwd (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-watc 80770 root rtd unknown /proc/80770/root (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-watc 80770 root txt unknown /proc/80770/exe (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-watc 80770 root NOFD /proc/80770/fd (opendir: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-comp 80771 root cwd unknown /proc/80771/cwd (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-comp 80771 root rtd unknown /proc/80771/root (readlink: Permission denied) 2024-04-22T06:40:17.777 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-comp 80771 root txt unknown /proc/80771/exe (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-comp 80771 root NOFD /proc/80771/fd (opendir: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81069 root cwd unknown /proc/81069/cwd (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81069 root rtd unknown /proc/81069/root (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81069 root txt unknown /proc/81069/exe (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81069 root NOFD /proc/81069/fd (opendir: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81129 root cwd unknown /proc/81129/cwd (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81129 root rtd unknown /proc/81129/root (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81129 root txt unknown /proc/81129/exe (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sshd 81129 root NOFD /proc/81129/fd (opendir: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 83648 root cwd unknown /proc/83648/cwd (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 83648 root rtd unknown /proc/83648/root (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 83648 root txt unknown /proc/83648/exe (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 83648 root NOFD /proc/83648/fd (opendir: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:daemon-he 83669 root cwd unknown /proc/83669/cwd (readlink: Permission denied) 2024-04-22T06:40:17.778 INFO:tasks.workunit.client.0.smithi105.stdout:daemon-he 83669 root rtd unknown /proc/83669/root (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:daemon-he 83669 root txt unknown /proc/83669/exe (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:daemon-he 83669 root NOFD /proc/83669/fd (opendir: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:bash 83686 root cwd unknown /proc/83686/cwd (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:bash 83686 root rtd unknown /proc/83686/root (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:bash 83686 root txt unknown /proc/83686/exe (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:bash 83686 root NOFD /proc/83686/fd (opendir: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 85812 root cwd unknown /proc/85812/cwd (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 85812 root rtd unknown /proc/85812/root (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 85812 root txt unknown /proc/85812/exe (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/3 85812 root NOFD /proc/85812/fd (opendir: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 86911 root cwd unknown /proc/86911/cwd (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 86911 root rtd unknown /proc/86911/root (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 86911 root txt unknown /proc/86911/exe (readlink: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:kworker/4 86911 root NOFD /proc/86911/fd (opendir: Permission denied) 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu cwd DIR 0,115 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:40:17.779 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu rtd DIR 8,1 4096 2 / 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu txt REG 8,1 41144 7580 /usr/bin/timeout 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 2586930 16078 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 2592488 7107 /usr/lib64/libc.so.6 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 346132 28859 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 54 16081 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 3284 28786 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 286 28783 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 57 26536 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 34 28836 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 77 16080 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 167 28778 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 59 28785 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 23 28782 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 26988 27963 /usr/lib64/gconv/gconv-modules.cache 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 867384 11866 /usr/lib64/ld-linux-x86-64.so.2 2024-04-22T06:40:17.780 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu mem REG 8,1 369 28781 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu 0r FIFO 0,13 0t0 235933 pipe 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu 1w FIFO 0,13 0t0 235934 pipe 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87217 ubuntu 2w FIFO 0,13 0t0 235935 pipe 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu cwd DIR 0,115 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu rtd DIR 8,1 4096 2 / 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu txt REG 8,1 1389072 7149 /usr/bin/bash 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 2586930 16078 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 2592488 7107 /usr/lib64/libc.so.6 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 346132 28859 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 54 16081 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 3284 28786 /usr/lib/locale/en_US.utf8/LC_TIME 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 286 28783 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 57 26536 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 34 28836 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-04-22T06:40:17.781 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 77 16080 /usr/lib/locale/en_US.utf8/LC_NAME 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 167 28778 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 59 28785 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 195080 7147 /usr/lib64/libtinfo.so.6.2 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 23 28782 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 26988 27963 /usr/lib64/gconv/gconv-modules.cache 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 867384 11866 /usr/lib64/ld-linux-x86-64.so.2 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu mem REG 8,1 369 28781 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu 0r FIFO 0,13 0t0 235933 pipe 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu 1w FIFO 0,13 0t0 235934 pipe 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu 2w FIFO 0,13 0t0 235935 pipe 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:fsync-tes 87236 ubuntu 255r REG 8,1 207 530736 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 87401 root cwd unknown /proc/87401/cwd (readlink: Permission denied) 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 87401 root rtd unknown /proc/87401/root (readlink: Permission denied) 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 87401 root txt unknown /proc/87401/exe (readlink: Permission denied) 2024-04-22T06:40:17.782 INFO:tasks.workunit.client.0.smithi105.stdout:sudo 87401 root NOFD /proc/87401/fd (opendir: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87418 root cwd unknown /proc/87418/cwd (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87418 root rtd unknown /proc/87418/root (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87418 root txt unknown /proc/87418/exe (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:timeout 87418 root NOFD /proc/87418/fd (opendir: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 root cwd unknown /proc/87420/cwd (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 root rtd unknown /proc/87420/root (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 root txt unknown /proc/87420/exe (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 root NOFD /proc/87420/fd (opendir: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87424 log root cwd unknown /proc/87420/task/87424/cwd (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87424 log root rtd unknown /proc/87420/task/87424/root (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87424 log root txt unknown /proc/87420/task/87424/exe (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87424 log root NOFD /proc/87420/task/87424/fd (opendir: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87425 msgr-work root cwd unknown /proc/87420/task/87425/cwd (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87425 msgr-work root rtd unknown /proc/87420/task/87425/root (readlink: Permission denied) 2024-04-22T06:40:17.783 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87425 msgr-work root txt unknown /proc/87420/task/87425/exe (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87425 msgr-work root NOFD /proc/87420/task/87425/fd (opendir: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87426 msgr-work root cwd unknown /proc/87420/task/87426/cwd (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87426 msgr-work root rtd unknown /proc/87420/task/87426/root (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87426 msgr-work root txt unknown /proc/87420/task/87426/exe (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87426 msgr-work root NOFD /proc/87420/task/87426/fd (opendir: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87427 msgr-work root cwd unknown /proc/87420/task/87427/cwd (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87427 msgr-work root rtd unknown /proc/87420/task/87427/root (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87427 msgr-work root txt unknown /proc/87420/task/87427/exe (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87427 msgr-work root NOFD /proc/87420/task/87427/fd (opendir: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87431 service root cwd unknown /proc/87420/task/87431/cwd (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87431 service root rtd unknown /proc/87420/task/87431/root (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87431 service root txt unknown /proc/87420/task/87431/exe (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87431 service root NOFD /proc/87420/task/87431/fd (opendir: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87432 io_contex root cwd unknown /proc/87420/task/87432/cwd (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87432 io_contex root rtd unknown /proc/87420/task/87432/root (readlink: Permission denied) 2024-04-22T06:40:17.784 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87432 io_contex root txt unknown /proc/87420/task/87432/exe (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87432 io_contex root NOFD /proc/87420/task/87432/fd (opendir: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87433 io_contex root cwd unknown /proc/87420/task/87433/cwd (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87433 io_contex root rtd unknown /proc/87420/task/87433/root (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87433 io_contex root txt unknown /proc/87420/task/87433/exe (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87433 io_contex root NOFD /proc/87420/task/87433/fd (opendir: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87434 ceph_time root cwd unknown /proc/87420/task/87434/cwd (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87434 ceph_time root rtd unknown /proc/87420/task/87434/root (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87434 ceph_time root txt unknown /proc/87420/task/87434/exe (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87434 ceph_time root NOFD /proc/87420/task/87434/fd (opendir: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87435 ms_dispat root cwd unknown /proc/87420/task/87435/cwd (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87435 ms_dispat root rtd unknown /proc/87420/task/87435/root (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87435 ms_dispat root txt unknown /proc/87420/task/87435/exe (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87435 ms_dispat root NOFD /proc/87420/task/87435/fd (opendir: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87436 ms_local root cwd unknown /proc/87420/task/87436/cwd (readlink: Permission denied) 2024-04-22T06:40:17.785 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87436 ms_local root rtd unknown /proc/87420/task/87436/root (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87436 ms_local root txt unknown /proc/87420/task/87436/exe (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87436 ms_local root NOFD /proc/87420/task/87436/fd (opendir: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87437 safe_time root cwd unknown /proc/87420/task/87437/cwd (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87437 safe_time root rtd unknown /proc/87420/task/87437/root (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87437 safe_time root txt unknown /proc/87420/task/87437/exe (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87437 safe_time root NOFD /proc/87420/task/87437/fd (opendir: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87438 safe_time root cwd unknown /proc/87420/task/87438/cwd (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87438 safe_time root rtd unknown /proc/87420/task/87438/root (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87438 safe_time root txt unknown /proc/87420/task/87438/exe (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87438 safe_time root NOFD /proc/87420/task/87438/fd (opendir: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87439 io_contex root cwd unknown /proc/87420/task/87439/cwd (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87439 io_contex root rtd unknown /proc/87420/task/87439/root (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87439 io_contex root txt unknown /proc/87420/task/87439/exe (readlink: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87439 io_contex root NOFD /proc/87420/task/87439/fd (opendir: Permission denied) 2024-04-22T06:40:17.786 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87440 io_contex root cwd unknown /proc/87420/task/87440/cwd (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87440 io_contex root rtd unknown /proc/87420/task/87440/root (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87440 io_contex root txt unknown /proc/87420/task/87440/exe (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87440 io_contex root NOFD /proc/87420/task/87440/fd (opendir: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87444 ceph_time root cwd unknown /proc/87420/task/87444/cwd (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87444 ceph_time root rtd unknown /proc/87420/task/87444/root (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87444 ceph_time root txt unknown /proc/87420/task/87444/exe (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87444 ceph_time root NOFD /proc/87420/task/87444/fd (opendir: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87445 fn_anonym root cwd unknown /proc/87420/task/87445/cwd (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87445 fn_anonym root rtd unknown /proc/87420/task/87445/root (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87445 fn_anonym root txt unknown /proc/87420/task/87445/exe (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87445 fn_anonym root NOFD /proc/87420/task/87445/fd (opendir: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87446 safe_time root cwd unknown /proc/87420/task/87446/cwd (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87446 safe_time root rtd unknown /proc/87420/task/87446/root (readlink: Permission denied) 2024-04-22T06:40:17.787 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87446 safe_time root txt unknown /proc/87420/task/87446/exe (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87446 safe_time root NOFD /proc/87420/task/87446/fd (opendir: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87447 fn_anonym root cwd unknown /proc/87420/task/87447/cwd (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87447 fn_anonym root rtd unknown /proc/87420/task/87447/root (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87447 fn_anonym root txt unknown /proc/87420/task/87447/exe (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87447 fn_anonym root NOFD /proc/87420/task/87447/fd (opendir: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87448 flusher root cwd unknown /proc/87420/task/87448/cwd (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87448 flusher root rtd unknown /proc/87420/task/87448/root (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87448 flusher root txt unknown /proc/87420/task/87448/exe (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87448 flusher root NOFD /proc/87420/task/87448/fd (opendir: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87449 ms_dispat root cwd unknown /proc/87420/task/87449/cwd (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87449 ms_dispat root rtd unknown /proc/87420/task/87449/root (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87449 ms_dispat root txt unknown /proc/87420/task/87449/exe (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87449 ms_dispat root NOFD /proc/87420/task/87449/fd (opendir: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87450 ms_local root cwd unknown /proc/87420/task/87450/cwd (readlink: Permission denied) 2024-04-22T06:40:17.788 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87450 ms_local root rtd unknown /proc/87420/task/87450/root (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87450 ms_local root txt unknown /proc/87420/task/87450/exe (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87450 ms_local root NOFD /proc/87420/task/87450/fd (opendir: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87451 safe_time root cwd unknown /proc/87420/task/87451/cwd (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87451 safe_time root rtd unknown /proc/87420/task/87451/root (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87451 safe_time root txt unknown /proc/87420/task/87451/exe (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph 87420 87451 safe_time root NOFD /proc/87420/task/87451/fd (opendir: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 87520 root cwd unknown /proc/87520/cwd (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 87520 root rtd unknown /proc/87520/root (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 87520 root txt unknown /proc/87520/exe (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:conmon 87520 root NOFD /proc/87520/fd (opendir: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 87522 root cwd unknown /proc/87522/cwd (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 87522 root rtd unknown /proc/87522/root (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 87522 root txt unknown /proc/87522/exe (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:podman-in 87522 root NOFD /proc/87522/fd (opendir: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 ceph cwd unknown /proc/87524/cwd (readlink: Permission denied) 2024-04-22T06:40:17.789 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 ceph rtd unknown /proc/87524/root (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 ceph txt unknown /proc/87524/exe (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 ceph NOFD /proc/87524/fd (opendir: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87525 log ceph cwd unknown /proc/87524/task/87525/cwd (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87525 log ceph rtd unknown /proc/87524/task/87525/root (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87525 log ceph txt unknown /proc/87524/task/87525/exe (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87525 log ceph NOFD /proc/87524/task/87525/fd (opendir: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87527 msgr-work ceph cwd unknown /proc/87524/task/87527/cwd (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87527 msgr-work ceph rtd unknown /proc/87524/task/87527/root (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87527 msgr-work ceph txt unknown /proc/87524/task/87527/exe (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87527 msgr-work ceph NOFD /proc/87524/task/87527/fd (opendir: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87528 msgr-work ceph cwd unknown /proc/87524/task/87528/cwd (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87528 msgr-work ceph rtd unknown /proc/87524/task/87528/root (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87528 msgr-work ceph txt unknown /proc/87524/task/87528/exe (readlink: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87528 msgr-work ceph NOFD /proc/87524/task/87528/fd (opendir: Permission denied) 2024-04-22T06:40:17.790 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87529 msgr-work ceph cwd unknown /proc/87524/task/87529/cwd (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87529 msgr-work ceph rtd unknown /proc/87524/task/87529/root (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87529 msgr-work ceph txt unknown /proc/87524/task/87529/exe (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87529 msgr-work ceph NOFD /proc/87524/task/87529/fd (opendir: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87533 service ceph cwd unknown /proc/87524/task/87533/cwd (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87533 service ceph rtd unknown /proc/87524/task/87533/root (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87533 service ceph txt unknown /proc/87524/task/87533/exe (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87533 service ceph NOFD /proc/87524/task/87533/fd (opendir: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87534 admin_soc ceph cwd unknown /proc/87524/task/87534/cwd (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87534 admin_soc ceph rtd unknown /proc/87524/task/87534/root (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87534 admin_soc ceph txt unknown /proc/87524/task/87534/exe (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87534 admin_soc ceph NOFD /proc/87524/task/87534/fd (opendir: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87535 signal_ha ceph cwd unknown /proc/87524/task/87535/cwd (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87535 signal_ha ceph rtd unknown /proc/87524/task/87535/root (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87535 signal_ha ceph txt unknown /proc/87524/task/87535/exe (readlink: Permission denied) 2024-04-22T06:40:17.791 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87535 signal_ha ceph NOFD /proc/87524/task/87535/fd (opendir: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87536 io_contex ceph cwd unknown /proc/87524/task/87536/cwd (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87536 io_contex ceph rtd unknown /proc/87524/task/87536/root (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87536 io_contex ceph txt unknown /proc/87524/task/87536/exe (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87536 io_contex ceph NOFD /proc/87524/task/87536/fd (opendir: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87537 io_contex ceph cwd unknown /proc/87524/task/87537/cwd (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87537 io_contex ceph rtd unknown /proc/87524/task/87537/root (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87537 io_contex ceph txt unknown /proc/87524/task/87537/exe (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87537 io_contex ceph NOFD /proc/87524/task/87537/fd (opendir: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87538 ms_dispat ceph cwd unknown /proc/87524/task/87538/cwd (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87538 ms_dispat ceph rtd unknown /proc/87524/task/87538/root (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87538 ms_dispat ceph txt unknown /proc/87524/task/87538/exe (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87538 ms_dispat ceph NOFD /proc/87524/task/87538/fd (opendir: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87539 ms_local ceph cwd unknown /proc/87524/task/87539/cwd (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87539 ms_local ceph rtd unknown /proc/87524/task/87539/root (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87539 ms_local ceph txt unknown /proc/87524/task/87539/exe (readlink: Permission denied) 2024-04-22T06:40:17.792 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87539 ms_local ceph NOFD /proc/87524/task/87539/fd (opendir: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87540 safe_time ceph cwd unknown /proc/87524/task/87540/cwd (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87540 safe_time ceph rtd unknown /proc/87524/task/87540/root (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87540 safe_time ceph txt unknown /proc/87524/task/87540/exe (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87540 safe_time ceph NOFD /proc/87524/task/87540/fd (opendir: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87541 safe_time ceph cwd unknown /proc/87524/task/87541/cwd (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87541 safe_time ceph rtd unknown /proc/87524/task/87541/root (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87541 safe_time ceph txt unknown /proc/87524/task/87541/exe (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87541 safe_time ceph NOFD /proc/87524/task/87541/fd (opendir: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87542 ceph-mds ceph cwd unknown /proc/87524/task/87542/cwd (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87542 ceph-mds ceph rtd unknown /proc/87524/task/87542/root (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87542 ceph-mds ceph txt unknown /proc/87524/task/87542/exe (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87542 ceph-mds ceph NOFD /proc/87524/task/87542/fd (opendir: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87543 safe_time ceph cwd unknown /proc/87524/task/87543/cwd (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87543 safe_time ceph rtd unknown /proc/87524/task/87543/root (readlink: Permission denied) 2024-04-22T06:40:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87543 safe_time ceph txt unknown /proc/87524/task/87543/exe (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87543 safe_time ceph NOFD /proc/87524/task/87543/fd (opendir: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87545 ceph_time ceph cwd unknown /proc/87524/task/87545/cwd (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87545 ceph_time ceph rtd unknown /proc/87524/task/87545/root (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87545 ceph_time ceph txt unknown /proc/87524/task/87545/exe (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87545 ceph_time ceph NOFD /proc/87524/task/87545/fd (opendir: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87548 OpHistory ceph cwd unknown /proc/87524/task/87548/cwd (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87548 OpHistory ceph rtd unknown /proc/87524/task/87548/root (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87548 OpHistory ceph txt unknown /proc/87524/task/87548/exe (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87548 OpHistory ceph NOFD /proc/87524/task/87548/fd (opendir: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87549 ms_dispat ceph cwd unknown /proc/87524/task/87549/cwd (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87549 ms_dispat ceph rtd unknown /proc/87524/task/87549/root (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87549 ms_dispat ceph txt unknown /proc/87524/task/87549/exe (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87549 ms_dispat ceph NOFD /proc/87524/task/87549/fd (opendir: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87550 ms_dispat ceph cwd unknown /proc/87524/task/87550/cwd (readlink: Permission denied) 2024-04-22T06:40:17.794 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87550 ms_dispat ceph rtd unknown /proc/87524/task/87550/root (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87550 ms_dispat ceph txt unknown /proc/87524/task/87550/exe (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87550 ms_dispat ceph NOFD /proc/87524/task/87550/fd (opendir: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87551 mds_rank_ ceph cwd unknown /proc/87524/task/87551/cwd (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87551 mds_rank_ ceph rtd unknown /proc/87524/task/87551/root (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87551 mds_rank_ ceph txt unknown /proc/87524/task/87551/exe (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87551 mds_rank_ ceph NOFD /proc/87524/task/87551/fd (opendir: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87552 PQ_Finish ceph cwd unknown /proc/87524/task/87552/cwd (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87552 PQ_Finish ceph rtd unknown /proc/87524/task/87552/root (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87552 PQ_Finish ceph txt unknown /proc/87524/task/87552/exe (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87552 PQ_Finish ceph NOFD /proc/87524/task/87552/fd (opendir: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87553 safe_time ceph cwd unknown /proc/87524/task/87553/cwd (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87553 safe_time ceph rtd unknown /proc/87524/task/87553/root (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87553 safe_time ceph txt unknown /proc/87524/task/87553/exe (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87553 safe_time ceph NOFD /proc/87524/task/87553/fd (opendir: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87554 MR_Finish ceph cwd unknown /proc/87524/task/87554/cwd (readlink: Permission denied) 2024-04-22T06:40:17.795 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87554 MR_Finish ceph rtd unknown /proc/87524/task/87554/root (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87554 MR_Finish ceph txt unknown /proc/87524/task/87554/exe (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87554 MR_Finish ceph NOFD /proc/87524/task/87554/fd (opendir: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87556 md_submit ceph cwd unknown /proc/87524/task/87556/cwd (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87556 md_submit ceph rtd unknown /proc/87524/task/87556/root (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87556 md_submit ceph txt unknown /proc/87524/task/87556/exe (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:ceph-mds 87524 87556 md_submit ceph NOFD /proc/87524/task/87556/fd (opendir: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:sleep 88145 root cwd unknown /proc/88145/cwd (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:sleep 88145 root rtd unknown /proc/88145/root (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:sleep 88145 root txt unknown /proc/88145/exe (readlink: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:sleep 88145 root NOFD /proc/88145/fd (opendir: Permission denied) 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu cwd DIR 0,115 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu rtd DIR 8,1 4096 2 / 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu txt REG 8,1 179232 9797 /usr/bin/lsof 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 4487176 7490 /usr/lib64/libcrypto.so.3.0.7 2024-04-22T06:40:17.796 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 346132 28859 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 102672 7169 /usr/lib64/libz.so.1.2.11 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 70928 7116 /usr/lib64/libresolv.so.2 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 66192 8888 /usr/lib64/libkrb5support.so.0.1 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 906256 8886 /usr/lib64/libkrb5.so.3.3 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 358448 8876 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 636848 7222 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 2592488 7107 /usr/lib64/libc.so.6 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 23992 7243 /usr/lib64/libkeyutils.so.1.10 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 23872 7193 /usr/lib64/libcom_err.so.2.1 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 98992 8880 /usr/lib64/libk5crypto.so.3.1 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 196224 8914 /usr/lib64/libtirpc.so.3.0.0 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 175752 7307 /usr/lib64/libselinux.so.1 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 26988 27963 /usr/lib64/gconv/gconv-modules.cache 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu mem REG 8,1 867384 11866 /usr/lib64/ld-linux-x86-64.so.2 2024-04-22T06:40:17.797 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 0r FIFO 0,13 0t0 235933 pipe 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 1w FIFO 0,13 0t0 235934 pipe 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 2w FIFO 0,13 0t0 235935 pipe 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 3r DIR 0,20 0 1 /proc 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 4r DIR 0,20 0 237214 /proc/88147/fd 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 5w FIFO 0,13 0t0 237219 pipe 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88147 ubuntu 6r FIFO 0,13 0t0 237220 pipe 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu cwd DIR 0,115 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu rtd DIR 8,1 4096 2 / 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu txt REG 8,1 179232 9797 /usr/bin/lsof 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 4487176 7490 /usr/lib64/libcrypto.so.3.0.7 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 346132 28859 /usr/lib/locale/C.utf8/LC_CTYPE 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 102672 7169 /usr/lib64/libz.so.1.2.11 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 70928 7116 /usr/lib64/libresolv.so.2 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 66192 8888 /usr/lib64/libkrb5support.so.0.1 2024-04-22T06:40:17.798 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 906256 8886 /usr/lib64/libkrb5.so.3.3 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 358448 8876 /usr/lib64/libgssapi_krb5.so.2.2 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 636848 7222 /usr/lib64/libpcre2-8.so.0.11.0 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 2592488 7107 /usr/lib64/libc.so.6 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 23992 7243 /usr/lib64/libkeyutils.so.1.10 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 23872 7193 /usr/lib64/libcom_err.so.2.1 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 98992 8880 /usr/lib64/libk5crypto.so.3.1 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 196224 8914 /usr/lib64/libtirpc.so.3.0.0 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 175752 7307 /usr/lib64/libselinux.so.1 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 26988 27963 /usr/lib64/gconv/gconv-modules.cache 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu mem REG 8,1 867384 11866 /usr/lib64/ld-linux-x86-64.so.2 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu 4r FIFO 0,13 0t0 237219 pipe 2024-04-22T06:40:17.799 INFO:tasks.workunit.client.0.smithi105.stdout:lsof 88148 ubuntu 7w FIFO 0,13 0t0 237220 pipe 2024-04-22T06:40:17.799 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-22T06:40:17.800 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-22T06:40:17.842 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-04-22T06:40:17.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-22T06:40:18.149 DEBUG:teuthology.parallel:result is None 2024-04-22T06:40:18.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:40:18.179 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-22T06:40:18.179 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-22T06:40:18.190 INFO:tasks.quiescer:joining Quiescers 2024-04-22T06:40:18.190 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:40:18.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 -- 172.21.15.105:0/1539148252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f959810b210 msgr2=0x7f9598113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 --2- 172.21.15.105:0/1539148252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f959810b210 0x7f9598113960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9588009860 tx=0x7f958802bb70 comp rx=0 tx=0).stop 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 -- 172.21.15.105:0/1539148252 shutdown_connections 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 --2- 172.21.15.105:0/1539148252 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9598113ea0 0x7f9598116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 --2- 172.21.15.105:0/1539148252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f959810b210 0x7f9598113960 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 --2- 172.21.15.105:0/1539148252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f959810acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 -- 172.21.15.105:0/1539148252 >> 172.21.15.105:0/1539148252 conn(0x7f959806e410 msgr2=0x7f95980719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 -- 172.21.15.105:0/1539148252 shutdown_connections 2024-04-22T06:40:18.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.387+0000 7f959fa4e640 1 -- 172.21.15.105:0/1539148252 wait complete. 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 Processor -- start 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 -- start start 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f959810b210 0x7f95981a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9598113ea0 0x7f959819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f95981187b0 con 0x7f9598113ea0 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959d7ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959dfeb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9598113ea0 0x7f959819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959d7ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:44008/0 (socket says 172.21.15.105:44008) 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959d7ea640 1 -- 172.21.15.105:0/3273414407 learned_addr learned my addr 172.21.15.105:0/3273414407 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9598118630 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.388+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9598118930 con 0x7f959810b210 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959cfe9640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f959810b210 0x7f95981a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 -- 172.21.15.105:0/3273414407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f959810b210 msgr2=0x7f95981a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f959810b210 0x7f95981a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 -- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9598113ea0 msgr2=0x7f959819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9598113ea0 0x7f959819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959819f430 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959d7ea640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f959400bc70 tx=0x7f9594008600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959401b070 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9594008d10 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959819f720 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594015600 con 0x7f959810a8f0 2024-04-22T06:40:18.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.389+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95981ab9e0 con 0x7f959810a8f0 2024-04-22T06:40:18.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.391+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c005290 con 0x7f959810a8f0 2024-04-22T06:40:18.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.392+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f959401e070 con 0x7f959810a8f0 2024-04-22T06:40:18.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.392+0000 7f958e7fc640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 0x7f95840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.393+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f95940d0550 con 0x7f959810a8f0 2024-04-22T06:40:18.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.393+0000 7f959cfe9640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 0x7f95840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.393+0000 7f959cfe9640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 0x7f95840a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f958802c180 tx=0x7f95880058d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.398+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f959409c360 con 0x7f959810a8f0 2024-04-22T06:40:18.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.627+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f957c005850 con 0x7f959810a8f0 2024-04-22T06:40:18.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.628+0000 7f958e7fc640 1 -- 172.21.15.105:0/3273414407 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11950 (secure 0 0 0) 0x7f95940a0660 con 0x7f959810a8f0 2024-04-22T06:40:18.628 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:40:18.628 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:39:31.042056+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":25432,"mds_2":25426,"mds_3":25415,"mds_4":16080},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16080":{"gid":16080,"name":"a","rank":4,"incarnation":41,"state":"up:active","state_seq":17,"addr":"172.21.15.105:6841/85174468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":85174468},{"type":"v1","addr":"172.21.15.105:6841","nonce":85174468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16398":{"gid":16398,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6839/798740274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":798740274},{"type":"v1","addr":"172.21.15.105:6839","nonce":798740274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25415":{"gid":25415,"name":"h","rank":3,"incarnation":39,"state":"up:active","state_seq":17,"addr":"172.21.15.112:6837/1826997924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":1826997924},{"type":"v1","addr":"172.21.15.112:6837","nonce":1826997924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25426":{"gid":25426,"name":"i","rank":2,"incarnation":45,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6837/678281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":678281724},{"type":"v1","addr":"172.21.15.177:6837","nonce":678281724}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25432":{"gid":25432,"name":"c","rank":1,"incarnation":43,"state":"up:active","state_seq":17,"addr":"172.21.15.177:6839/3064875873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3064875873},{"type":"v1","addr":"172.21.15.177:6839","nonce":3064875873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25685":{"gid":25685,"name":"e","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/3361835468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":3361835468},{"type":"v1","addr":"172.21.15.112:6833","nonce":3361835468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25690":{"gid":25690,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6833/3918665016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3918665016},{"type":"v1","addr":"172.21.15.177:6833","nonce":3918665016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1127626841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1127626841},{"type":"v1","addr":"172.21.15.112:6835","nonce":1127626841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24500,16080,25432,25426,25415]},"id":1}]} 2024-04-22T06:40:18.628 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 49 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 msgr2=0x7f95840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 0x7f95840a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f958802c180 tx=0x7f95880058d0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 msgr2=0x7f95981a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f959400bc70 tx=0x7f9594008600 comp rx=0 tx=0).stop 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 shutdown_connections 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f95840a3d20 0x7f95840a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9598113ea0 0x7f959819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f959810b210 0x7f95981a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 --2- 172.21.15.105:0/3273414407 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f959810a8f0 0x7f95981a4a60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 >> 172.21.15.105:0/3273414407 conn(0x7f959806e410 msgr2=0x7f9598070080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 shutdown_connections 2024-04-22T06:40:18.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.630+0000 7f959fa4e640 1 -- 172.21.15.105:0/3273414407 wait complete. 2024-04-22T06:40:18.642 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24500 quiesce db --cancel --all' 2024-04-22T06:40:18.642 DEBUG:teuthology.orchestra.run.smithi105:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24500 quiesce db --cancel --all 2024-04-22T06:40:18.765 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 -- 172.21.15.105:0/455386219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0102180 msgr2=0x7f5bb0102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 --2- 172.21.15.105:0/455386219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0102180 0x7f5bb0102560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009900 tx=0x7f5ba402bc30 comp rx=0 tx=0).stop 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 -- 172.21.15.105:0/455386219 shutdown_connections 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 --2- 172.21.15.105:0/455386219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb0107250 0x7f5bb010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 --2- 172.21.15.105:0/455386219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102b30 0x7f5bb0106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 --2- 172.21.15.105:0/455386219 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0102180 0x7f5bb0102560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 -- 172.21.15.105:0/455386219 >> 172.21.15.105:0/455386219 conn(0x7f5bb00fd9c0 msgr2=0x7f5bb00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 -- 172.21.15.105:0/455386219 shutdown_connections 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.750+0000 7f5bb7255640 1 -- 172.21.15.105:0/455386219 wait complete. 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 Processor -- start 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 -- start start 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 0x7f5bb019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 0x7f5bb019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01121b0 con 0x7f5bb01986a0 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0112030 con 0x7f5bb0107250 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb7255640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0112330 con 0x7f5bb0102180 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5baffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 0x7f5bb019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.768 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45196/0 (socket says 172.21.15.105:45196) 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 -- 172.21.15.105:0/2270530059 learned_addr learned my addr 172.21.15.105:0/2270530059 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 0x7f5bb019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 -- 172.21.15.105:0/2270530059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 msgr2=0x7f5bb019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 0x7f5bb019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 -- 172.21.15.105:0/2270530059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 msgr2=0x7f5bb019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 0x7f5bb019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 -- 172.21.15.105:0/2270530059 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01992a0 con 0x7f5bb01986a0 2024-04-22T06:40:18.769 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 0x7f5bb019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5baffff640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 0x7f5bb019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.751+0000 7f5bb57f2640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800efd0 tx=0x7f5ba800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.752+0000 7f5badffb640 1 -- 172.21.15.105:0/2270530059 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8003b70 con 0x7f5bb01986a0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.752+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb01a5330 con 0x7f5bb01986a0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.752+0000 7f5badffb640 1 -- 172.21.15.105:0/2270530059 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba8003d10 con 0x7f5bb01986a0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.752+0000 7f5badffb640 1 -- 172.21.15.105:0/2270530059 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80108f0 con 0x7f5bb01986a0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.752+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb01a5730 con 0x7f5bb01986a0 2024-04-22T06:40:18.770 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5badffb640 1 -- 172.21.15.105:0/2270530059 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f5ba8004560 con 0x7f5bb01986a0 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 Processor -- start 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 -- start start 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f8b70 0x56100e0f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56100e05e000 con 0x56100e0f9520 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56100dffab20 con 0x56100e102680 2024-04-22T06:40:18.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.754+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56100e059fc0 con 0x56100e0f8b70 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5badffb640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 0x7f5ba00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb4ff1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb4ff1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45208/0 (socket says 172.21.15.105:45208) 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:44034/0 (socket says 172.21.15.105:44034) 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb4ff1640 1 -- 172.21.15.105:0/4015311500 learned_addr learned my addr 172.21.15.105:0/4015311500 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb57f2640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 0x7f5ba00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f8b70 0x56100e0f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5badffb640 1 -- 172.21.15.105:0/2270530059 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f5ba80d0eb0 con 0x7f5bb01986a0 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 -- 172.21.15.105:0/4015311500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f8b70 msgr2=0x56100e0f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f8b70 0x56100e0f8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 -- 172.21.15.105:0/4015311500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 msgr2=0x56100e102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 -- 172.21.15.105:0/4015311500 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100e014ea0 con 0x56100e102680 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5bb57f2640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 0x7f5ba00a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ba809cba0 tx=0x7f5ba8009f90 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5baffff640 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0199ea0 tx=0x7f5b9c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c784f88cb7a7775 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.755+0000 7f5b9affd640 1 -- 172.21.15.105:0/4015311500 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c011050 con 0x56100e102680 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5b9affd640 1 -- 172.21.15.105:0/4015311500 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c0054d0 con 0x56100e102680 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5b9affd640 1 -- 172.21.15.105:0/4015311500 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00fe80 con 0x56100e102680 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 -- 172.21.15.105:0/4015311500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 msgr2=0x56100e102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0199ea0 tx=0x7f5b9c005850 comp rx=0 tx=0).stop 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 -- 172.21.15.105:0/4015311500 shutdown_connections 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e102b00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f9520 0x56100e102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/4015311500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f8b70 0x56100e0f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 -- 172.21.15.105:0/4015311500 >> 172.21.15.105:0/4015311500 conn(0x56100e0f33e0 msgr2=0x56100e0f37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 -- 172.21.15.105:0/4015311500 shutdown_connections 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.756+0000 7f5bb9e47740 1 -- 172.21.15.105:0/4015311500 wait complete. 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 -- start start 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 Processor -- start 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e106120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56100e109600 con 0x56100e102680 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56100dff8510 con 0x56100e0f8b70 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x56100e109780 con 0x56100e0f9520 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb9e47740 10 client.0 _ll_register_callbacks cb 0x56100dff7f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5baffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb4ff1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5baffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45220/0 (socket says 172.21.15.105:45220) 2024-04-22T06:40:18.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5bb4ff1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.105:58756/0 (socket says 172.21.15.105:58756) 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5baffff640 1 -- 172.21.15.105:0/2918996394 learned_addr learned my addr 172.21.15.105:0/2918996394 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.757+0000 7f5baffff640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e106120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 -- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 msgr2=0x56100e106120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e106120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 -- 172.21.15.105:0/2918996394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 msgr2=0x56100e106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 -- 172.21.15.105:0/2918996394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56100de3ca20 con 0x56100e0f9520 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5baffff640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x56100e0f7ee0 tx=0x7f5ba402ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80bc24fd7d52362 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.177:3300/0 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4045070 con 0x56100e0f9520 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mon.1 v2:172.21.15.177:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f5ba4005450 con 0x56100e0f9520 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9c0054d0 con 0x56100e0f9520 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5ba4005d50 con 0x56100e0f9520 2024-04-22T06:40:18.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb9e47740 10 client.25807 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56100de9d1b0 con 0x56100e0f9520 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mon.1 v2:172.21.15.177:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c0054d0 con 0x56100e0f9520 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.758+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mon.1 v2:172.21.15.177:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5ba40035f0 con 0x56100e0f9520 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 10 client.25807 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 10 client.25807 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 20 client.25807 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e116b80 0x56100e118fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 4 client.25807 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56100e06ba20 con 0x56100e116b80 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5baffff640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e116b80 0x56100e118fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5baffff640 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e116b80 0x56100e118fb0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8df27bd29d76df7e server_cookie=5e1e651542849fdb in_seq=0 out_seq=0 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.759+0000 7f5b8ffff640 10 client.25807 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5b8ffff640 1 -- 172.21.15.105:0/2918996394 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56100e06ba20 con 0x56100e116b80 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5b8ffff640 10 client.25807 handle_command_reply: tid=0 2024-04-22T06:40:18.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 client.25807 shutdown 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 msgr2=0x56100e106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x56100e0f7ee0 tx=0x7f5ba402ca90 comp rx=0 tx=0).stop 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 shutdown_connections 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e116b80 0x56100e118fb0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e106660 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/2918996394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5b8ffff640 0 client.25807 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 >> 172.21.15.105:0/2918996394 conn(0x56100e0f33e0 msgr2=0x56100e0f5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 shutdown_connections 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 1 -- 172.21.15.105:0/2918996394 wait complete. 2024-04-22T06:40:18.778 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.760+0000 7f5bb9e47740 20 client.25807 trim_cache size 0 max 16384 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 Processor -- start 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 -- start start 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e112530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e1050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.784 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01abfd0 con 0x56100e102680 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01adb10 con 0x56100e0f8b70 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01ac6a0 con 0x56100e0f9520 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e1050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.105:44058/0 (socket says 172.21.15.105:44058) 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 -- 172.21.15.105:0/1627549130 learned_addr learned my addr 172.21.15.105:0/1627549130 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 -- 172.21.15.105:0/1627549130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 msgr2=0x56100e112530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e112530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 -- 172.21.15.105:0/1627549130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 msgr2=0x56100e1050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.778+0000 7f5baffff640 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e1050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.779+0000 7f5baffff640 1 -- 172.21.15.105:0/1627549130 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100dd30310 con 0x56100e0f8b70 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.779+0000 7f5baffff640 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00fc30 tx=0x7f5b9c009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc14cef3042a01b server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.779+0000 7f5b9a7fc640 1 -- 172.21.15.105:0/1627549130 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c005450 con 0x56100e0f8b70 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.779+0000 7f5b9a7fc640 1 -- 172.21.15.105:0/1627549130 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c011720 con 0x56100e0f8b70 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.779+0000 7f5b9a7fc640 1 -- 172.21.15.105:0/1627549130 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c010690 con 0x56100e0f8b70 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- 172.21.15.105:0/1627549130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 msgr2=0x56100e108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00fc30 tx=0x7f5b9c009eb0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- 172.21.15.105:0/1627549130 shutdown_connections 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e102680 0x56100e1050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/1627549130 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e0f8b70 0x56100e108850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- 172.21.15.105:0/1627549130 >> 172.21.15.105:0/1627549130 conn(0x56100e0f33e0 msgr2=0x56100e0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- 172.21.15.105:0/1627549130 shutdown_connections 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- 172.21.15.105:0/1627549130 wait complete. 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- start start 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 Processor -- start 2024-04-22T06:40:18.786 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e10ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e10ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01ad440 con 0x56100e0f8b70 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01acd70 con 0x56100e102680 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb01aa010 con 0x56100e0f9520 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5bb9e47740 10 client.0 _ll_register_callbacks cb 0x7f5ba4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5baffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5baffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45250/0 (socket says 172.21.15.105:45250) 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.780+0000 7f5baffff640 1 -- 172.21.15.105:0/442044055 learned_addr learned my addr 172.21.15.105:0/442044055 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e10ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e10ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 -- 172.21.15.105:0/442044055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 msgr2=0x56100e10ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e10ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 -- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 msgr2=0x56100e10ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-22T06:40:18.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e10ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 -- 172.21.15.105:0/442044055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56100e10c210 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5baffff640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009c90 tx=0x7f5b9c00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6a85976f298f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5b8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c02f8d0 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mon.0 v2:172.21.15.105:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9099+0+0 (secure 0 0 0) 0x7f5b9c005450 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9c011720 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5b9c02f3d0 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5bb9e47740 10 client.16554 fetch_fsmap learned FSMap version 49 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56100e17db80 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.781+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mon.0 v2:172.21.15.105:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b9c011720 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mon.0 v2:172.21.15.105:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5b9c01d030 con 0x56100e0f8b70 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 10 client.16554 fetch_fsmap finished waiting for FSMap version 49 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 10 client.16554 resolve_mds: validated gid 24500 aka daemon mds.b 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 20 client.16554 populate_metadata read hostname 'smithi105' 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e17dfa0 0x56100e180390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:40:18.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 4 client.16554 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 --> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x56100e065890 con 0x56100e17dfa0 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e17dfa0 0x56100e180390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5bb4ff1640 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e17dfa0 0x56100e180390 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7d77a5867e87771 server_cookie=e99d549f09cd605 in_seq=0 out_seq=0 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.782+0000 7f5b8ffff640 10 client.16554 ms_handle_connect on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5b8ffff640 1 -- 172.21.15.105:0/442044055 <== mds.0 v2:172.21.15.112:6838/4112363218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x56100e065890 con 0x56100e17dfa0 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5b8ffff640 10 client.16554 handle_command_reply: tid=0 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 client.16554 shutdown 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 msgr2=0x56100e10ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009c90 tx=0x7f5b9c00ffd0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 shutdown_connections 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:6838/4112363218,v1:172.21.15.112:6839/4112363218] conn(0x56100e17dfa0 0x56100e180390 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56100e102680 0x56100e10ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x56100e0f9520 0x56100e10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5b8ffff640 0 client.16554 ms_handle_reset on v2:172.21.15.112:6838/4112363218 2024-04-22T06:40:18.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 --2- 172.21.15.105:0/442044055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56100e0f8b70 0x56100e10ea40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 >> 172.21.15.105:0/442044055 conn(0x56100e0f33e0 msgr2=0x56100e0f4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 shutdown_connections 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 1 -- 172.21.15.105:0/442044055 wait complete. 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.783+0000 7f5bb9e47740 20 client.16554 trim_cache size 0 max 16384 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 49, 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout: "leader": 24500, 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout: "set_version": 12, 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout: "sets": {} 2024-04-22T06:40:18.790 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:40:18.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 msgr2=0x7f5ba00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 0x7f5ba00a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5ba809cba0 tx=0x7f5ba8009f90 comp rx=0 tx=0).stop 2024-04-22T06:40:18.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 msgr2=0x7f5bb0198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800efd0 tx=0x7f5ba800c6a0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 shutdown_connections 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f5ba00a3d20 0x7f5ba00a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5bb01986a0 0x7f5bb0198b20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb0107250 0x7f5bb019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 --2- 172.21.15.105:0/2270530059 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5bb0102180 0x7f5bb019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 >> 172.21.15.105:0/2270530059 conn(0x7f5bb00fd9c0 msgr2=0x7f5bb00fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 shutdown_connections 2024-04-22T06:40:18.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:18.784+0000 7f5bb7255640 1 -- 172.21.15.105:0/2270530059 wait complete. 2024-04-22T06:40:18.794 INFO:tasks.quiescer:done joining Quiescers 2024-04-22T06:40:18.795 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-22T06:40:18.807 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:40:18.815 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-22T06:40:18.874 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-22T06:40:18.875 INFO:teuthology.task.exec:Executing custom commands... 2024-04-22T06:40:18.875 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:40:18.875 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-22T06:40:18.900 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-22T06:40:19.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:18 smithi105 ceph-mon[27916]: pgmap v91: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:19.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:18 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3273414407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:19.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:18 smithi112 ceph-mon[31236]: pgmap v91: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:19.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:18 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3273414407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:19.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:18 smithi177 ceph-mon[31346]: pgmap v91: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 256 op/s 2024-04-22T06:40:19.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:18 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3273414407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:40:20.744 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:40:20.744 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 systemd[1]: Stopped Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:20.744 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 systemd[1]: Starting Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:40:20.745 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 podman[64845]: 2024-04-22 06:40:20.598616181 +0000 UTC m=+0.018262368 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:40:20.954 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:20 smithi105 ceph-mon[27916]: pgmap v92: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.9 MiB/s wr, 551 op/s 2024-04-22T06:40:20.954 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:20 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:40:20.955 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:20 smithi105 systemd[1]: Stopped Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:20.955 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:20 smithi105 systemd[1]: Starting Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:40:20.955 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:20 smithi105 podman[88403]: 2024-04-22 06:40:20.810997083 +0000 UTC m=+0.018523451 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:40:20.955 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:20 smithi105 podman[88403]: 2024-04-22 06:40:20.95705386 +0000 UTC m=+0.164580214 container create 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, ceph=True) 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:20 smithi112 ceph-mon[31236]: pgmap v92: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.9 MiB/s wr, 551 op/s 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 podman[64845]: 2024-04-22 06:40:20.746965495 +0000 UTC m=+0.166611672 container create 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 podman[64845]: 2024-04-22 06:40:20.957987591 +0000 UTC m=+0.377633761 container init 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 podman[64845]: 2024-04-22 06:40:20.960795614 +0000 UTC m=+0.380441785 container start 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 ceph-mds[64863]: -- 172.21.15.112:0/1069060666 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55983cc21a40 con 0x55983cbf4800 2024-04-22T06:40:21.039 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: starting mds.h at 2024-04-22T06:40:21.040 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:21 smithi112 bash[64845]: 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 2024-04-22T06:40:21.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:20 smithi177 ceph-mon[31346]: pgmap v92: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 2.9 MiB/s wr, 551 op/s 2024-04-22T06:40:21.303 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:40:21 smithi112 systemd[1]: Started Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:21.490 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: Stopped Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: Starting Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: Stopped Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:21 smithi177 systemd[1]: Starting Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:40:21.491 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65987]: 2024-04-22 06:40:21.392172739 +0000 UTC m=+0.018991900 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 podman[88403]: 2024-04-22 06:40:21.227011056 +0000 UTC m=+0.434537407 container init 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 podman[88403]: 2024-04-22 06:40:21.230586252 +0000 UTC m=+0.438112608 container start 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mds[88422]: -- 172.21.15.105:0/1684676771 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560f5e79ba40 con 0x560f5e76f000 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: starting mds.a at 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 bash[88403]: 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 2024-04-22T06:40:21.668 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:40:21 smithi105 systemd[1]: Started Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:21.763 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65988]: 2024-04-22 06:40:21.398453311 +0000 UTC m=+0.021558711 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:40:21.763 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65987]: 2024-04-22 06:40:21.530114711 +0000 UTC m=+0.156933840 container create be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5) 2024-04-22T06:40:22.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: daemon mds.h restarted 2024-04-22T06:40:22.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:40:22.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:40:22.083 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/482071778,v1:172.21.15.112:6837/482071778] up:boot 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: daemon mds.a restarted 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:resolve 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/3795610143,v1:172.21.15.105:6841/3795610143] up:boot 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:21 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-22T06:40:22.084 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65988]: 2024-04-22 06:40:21.765441363 +0000 UTC m=+0.388546748 container create 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:40:22.085 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65988]: 2024-04-22 06:40:21.985281821 +0000 UTC m=+0.608387206 container init 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:40:22.085 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:21 smithi177 podman[65988]: 2024-04-22 06:40:21.987875214 +0000 UTC m=+0.610980602 container start 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:40:22.085 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mds[66023]: -- 172.21.15.177:0/997638617 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564081a65a40 con 0x564081a39000 2024-04-22T06:40:22.085 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: starting mds.c at 2024-04-22T06:40:22.085 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:22 smithi177 bash[65988]: 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: daemon mds.h restarted 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/482071778,v1:172.21.15.112:6837/482071778] up:boot 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: daemon mds.a restarted 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:resolve 2024-04-22T06:40:22.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/3795610143,v1:172.21.15.105:6841/3795610143] up:boot 2024-04-22T06:40:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:40:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:40:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:40:22.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:21 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: daemon mds.h restarted 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: osdmap e101: 12 total, 12 up, 12 in 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/482071778,v1:172.21.15.112:6837/482071778] up:boot 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,4=a=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-22T06:40:22.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: daemon mds.a restarted 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: osdmap e102: 12 total, 12 up, 12 in 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:resolve 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/3795610143,v1:172.21.15.105:6841/3795610143] up:boot 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:40:22.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:21 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:40:22 smithi177 systemd[1]: Started Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 podman[65987]: 2024-04-22 06:40:22.20320173 +0000 UTC m=+0.830020860 container init be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 podman[65987]: 2024-04-22 06:40:22.20581908 +0000 UTC m=+0.832638218 container start be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mds[66044]: -- 172.21.15.177:0/2834590179 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56227c0d7a40 con 0x56227c0aa800 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: starting mds.i at 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 bash[65987]: be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c 2024-04-22T06:40:22.490 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:40:22 smithi177 systemd[1]: Started Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: pgmap v93: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 546 op/s 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: daemon mds.c restarted 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/2359784785,v1:172.21.15.177:6837/2359784785] up:boot 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: Replacing daemon mds.i as rank 2 with standby daemon mds.l 2024-04-22T06:40:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:40:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-22T06:40:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: fsmap cephfs:4/5 {0=b=up:active,2=i=up:active,3=e=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:40:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:40:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:40:23.169 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:22 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 3 up:replay 1 up:resolve 1 up:active 2024-04-22T06:40:23.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: pgmap v93: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 546 op/s 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: daemon mds.c restarted 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/2359784785,v1:172.21.15.177:6837/2359784785] up:boot 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: Replacing daemon mds.i as rank 2 with standby daemon mds.l 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: fsmap cephfs:4/5 {0=b=up:active,2=i=up:active,3=e=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:40:23.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:22 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 3 up:replay 1 up:resolve 1 up:active 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: pgmap v93: 97 pgs: 97 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.9 MiB/s wr, 546 op/s 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: daemon mds.c restarted 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: osdmap e103: 12 total, 12 up, 12 in 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/2359784785,v1:172.21.15.177:6837/2359784785] up:boot 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: Replacing daemon mds.i as rank 2 with standby daemon mds.l 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: fsmap cephfs:4/5 {0=b=up:active,2=i=up:active,3=e=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-22T06:40:23.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:40:23.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: osdmap e104: 12 total, 12 up, 12 in 2024-04-22T06:40:23.241 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:22 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 3 up:replay 1 up:resolve 1 up:active 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: pgmap v98: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:resolve 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:resolve 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:resolve 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6838/3491723308,v1:172.21.15.177:6839/3491723308] up:boot 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:40:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:24 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:40:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: pgmap v98: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:resolve 2024-04-22T06:40:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:resolve 2024-04-22T06:40:24.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:resolve 2024-04-22T06:40:24.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6838/3491723308,v1:172.21.15.177:6839/3491723308] up:boot 2024-04-22T06:40:24.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:40:24.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:24 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:40:24.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: pgmap v98: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:resolve 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:resolve 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:resolve 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6838/3491723308,v1:172.21.15.177:6839/3491723308] up:boot 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-04-22T06:40:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:24 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999973 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:reconnect 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:reconnect 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:reconnect 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:reconnect 2024-04-22T06:40:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:25 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:40:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999973 2024-04-22T06:40:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:40:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0 2024-04-22T06:40:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:reconnect 2024-04-22T06:40:25.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:reconnect 2024-04-22T06:40:25.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:reconnect 2024-04-22T06:40:25.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:reconnect 2024-04-22T06:40:25.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:25 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:40:25.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0.000999973 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: reconnect by client.15378 192.168.0.1:0/3678111489 after 0 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: reconnect by client.15834 172.21.15.105:0/3954931468 after 0 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:reconnect 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:reconnect 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:reconnect 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:reconnect 2024-04-22T06:40:25.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:25 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-04-22T06:40:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: pgmap v99: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:rejoin 2024-04-22T06:40:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:rejoin 2024-04-22T06:40:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:rejoin 2024-04-22T06:40:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:rejoin 2024-04-22T06:40:26.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:40:26.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:40:26.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:40:26.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-22T06:40:26.419 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:26 smithi105 ceph-mon[27916]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: pgmap v99: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:rejoin 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:rejoin 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:rejoin 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:rejoin 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:40:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:40:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:40:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-22T06:40:26.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:26 smithi112 ceph-mon[31236]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: pgmap v99: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 1.68k op/s 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:rejoin 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:rejoin 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:rejoin 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:rejoin 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-22T06:40:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:26 smithi177 ceph-mon[31346]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:active 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:active 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:active 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:active 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-04-22T06:40:27.391 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:27 smithi177 ceph-mon[31346]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-04-22T06:40:27.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:27 smithi105 ceph-mon[27916]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:40:27.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:40:27.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:40:27.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/798740274,v1:172.21.15.105:6839/798740274] up:active 2024-04-22T06:40:27.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/3918665016,v1:172.21.15.177:6833/3918665016] up:active 2024-04-22T06:40:27.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6834/1127626841,v1:172.21.15.112:6835/1127626841] up:active 2024-04-22T06:40:27.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/3361835468,v1:172.21.15.112:6833/3361835468] up:active 2024-04-22T06:40:27.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-04-22T06:40:27.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-mon[31236]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-22T06:40:28.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:28 smithi112 ceph-mon[31236]: pgmap v100: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-22T06:40:28.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:40:27.956+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-22T06:40:28.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:40:27.956+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-22T06:40:28.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:40:27.956+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-22T06:40:28.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:40:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:40:27.956+0000 7f75741b7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-22T06:40:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:28 smithi105 ceph-mon[27916]: pgmap v100: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-22T06:40:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:28 smithi177 ceph-mon[31346]: pgmap v100: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-22T06:40:30.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:30 smithi105 ceph-mon[27916]: pgmap v101: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 2.60k op/s 2024-04-22T06:40:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:30 smithi112 ceph-mon[31236]: pgmap v101: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 2.60k op/s 2024-04-22T06:40:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:30 smithi177 ceph-mon[31346]: pgmap v101: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 18 MiB/s wr, 2.60k op/s 2024-04-22T06:40:30.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:30.652+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:30.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:30.653+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:40:30.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:30.653+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:40:30.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:30.653+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:40:30.651 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:30.653+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:40:32.315 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:32 smithi112 ceph-mon[31236]: pgmap v102: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 2.24k op/s 2024-04-22T06:40:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:32 smithi105 ceph-mon[27916]: pgmap v102: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 2.24k op/s 2024-04-22T06:40:32.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:32 smithi177 ceph-mon[31346]: pgmap v102: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 2.24k op/s 2024-04-22T06:40:34.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:34 smithi105 ceph-mon[27916]: pgmap v103: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.83k op/s 2024-04-22T06:40:34.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:34 smithi112 ceph-mon[31236]: pgmap v103: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.83k op/s 2024-04-22T06:40:34.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:34 smithi177 ceph-mon[31346]: pgmap v103: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 1.83k op/s 2024-04-22T06:40:36.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:36 smithi105 ceph-mon[27916]: pgmap v104: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:36.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:36 smithi112 ceph-mon[31236]: pgmap v104: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:36.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:36 smithi177 ceph-mon[31346]: pgmap v104: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:38 smithi105 ceph-mon[27916]: pgmap v105: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:38 smithi112 ceph-mon[31236]: pgmap v105: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:38.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:38 smithi177 ceph-mon[31346]: pgmap v105: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:40.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:40 smithi105 ceph-mon[27916]: pgmap v106: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:40.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:40 smithi112 ceph-mon[31236]: pgmap v106: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:40.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:40 smithi177 ceph-mon[31346]: pgmap v106: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.8 MiB/s wr, 1.12k op/s 2024-04-22T06:40:42.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:42 smithi105 ceph-mon[27916]: pgmap v107: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:40:42.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:42 smithi112 ceph-mon[31236]: pgmap v107: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:40:42.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:42 smithi177 ceph-mon[31346]: pgmap v107: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-22T06:40:44.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:44 smithi105 ceph-mon[27916]: pgmap v108: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T06:40:44.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:44 smithi112 ceph-mon[31236]: pgmap v108: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T06:40:44.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:44 smithi177 ceph-mon[31346]: pgmap v108: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-22T06:40:45.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:45 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:45.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:45 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:45.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:45 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:40:45.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:45.667+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:40:45.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:45.667+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:40:45.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:45.667+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:40:45.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:45.667+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:40:45.666 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:40:45.667+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:40:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:46 smithi105 ceph-mon[27916]: pgmap v109: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:46.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:40:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:46 smithi112 ceph-mon[31236]: pgmap v109: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:40:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:46 smithi177 ceph-mon[31346]: pgmap v109: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:40:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:40:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:40:48.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:48 smithi105 ceph-mon[27916]: pgmap v110: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:48.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:48 smithi112 ceph-mon[31236]: pgmap v110: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:48.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:48 smithi177 ceph-mon[31346]: pgmap v110: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-22T06:40:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:50 smithi105 ceph-mon[27916]: pgmap v111: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-22T06:40:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:50 smithi112 ceph-mon[31236]: pgmap v111: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-22T06:40:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:50 smithi177 ceph-mon[31346]: pgmap v111: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-22T06:40:52.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:52 smithi105 ceph-mon[27916]: pgmap v112: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:52 smithi112 ceph-mon[31236]: pgmap v112: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:52.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:52 smithi177 ceph-mon[31346]: pgmap v112: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:54.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:54 smithi105 ceph-mon[27916]: pgmap v113: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:40:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:54 smithi112 ceph-mon[31236]: pgmap v113: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:40:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:54 smithi177 ceph-mon[31346]: pgmap v113: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:40:56.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:56 smithi105 ceph-mon[27916]: pgmap v114: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:56.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:56 smithi112 ceph-mon[31236]: pgmap v114: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:56.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:56 smithi177 ceph-mon[31346]: pgmap v114: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:58.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:40:58 smithi105 ceph-mon[27916]: pgmap v115: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:58.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:40:58 smithi112 ceph-mon[31236]: pgmap v115: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:40:58.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:40:58 smithi177 ceph-mon[31346]: pgmap v115: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:00.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:00 smithi105 ceph-mon[27916]: pgmap v116: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:00.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:00 smithi112 ceph-mon[31236]: pgmap v116: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:00.488 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:00 smithi177 ceph-mon[31346]: pgmap v116: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:00.677+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:00.677+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:41:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:00.677+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:41:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:00.677+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:41:00.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:00.677+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:41:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:02 smithi105 ceph-mon[27916]: pgmap v117: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:02 smithi112 ceph-mon[31236]: pgmap v117: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:02.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:02 smithi177 ceph-mon[31346]: pgmap v117: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:04 smithi105 ceph-mon[27916]: pgmap v118: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:04 smithi112 ceph-mon[31236]: pgmap v118: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:04.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:04 smithi177 ceph-mon[31346]: pgmap v118: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:06.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:06 smithi105 ceph-mon[27916]: pgmap v119: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:06.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:06 smithi112 ceph-mon[31236]: pgmap v119: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:06.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:06 smithi177 ceph-mon[31346]: pgmap v119: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:08.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:08 smithi105 ceph-mon[27916]: pgmap v120: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:08.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:08 smithi112 ceph-mon[31236]: pgmap v120: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:08 smithi177 ceph-mon[31346]: pgmap v120: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:10.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:10 smithi105 ceph-mon[27916]: pgmap v121: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:10.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:10 smithi112 ceph-mon[31236]: pgmap v121: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:10.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:10 smithi177 ceph-mon[31346]: pgmap v121: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:12.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:12 smithi105 ceph-mon[27916]: pgmap v122: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:12.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:12 smithi112 ceph-mon[31236]: pgmap v122: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:12 smithi177 ceph-mon[31346]: pgmap v122: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:41:14.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:14 smithi105 ceph-mon[27916]: pgmap v123: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:14.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:14 smithi112 ceph-mon[31236]: pgmap v123: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:14.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:14 smithi177 ceph-mon[31346]: pgmap v123: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:41:15.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:15.686+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:41:15.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:15.687+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-04-22T06:41:15.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:15.687+0000 7fd9f6ffd640 1 -- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 msgr2=0x564f1b111020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-22T06:41:15.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:15.687+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-22T06:41:15.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:41:15.687+0000 7fd9f6ffd640 1 --2- 172.21.15.105:0/2236332774 >> [v2:172.21.15.105:6838/256609249,v1:172.21.15.105:6839/256609249] conn(0x564f1b10ec30 0x564f1b111020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-22T06:41:16.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:16 smithi105 ceph-mon[27916]: pgmap v124: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:16.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:16 smithi112 ceph-mon[31236]: pgmap v124: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:16.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:16 smithi177 ceph-mon[31346]: pgmap v124: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:18 smithi105 ceph-mon[27916]: pgmap v125: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:18.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:18 smithi112 ceph-mon[31236]: pgmap v125: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:18.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:18 smithi177 ceph-mon[31346]: pgmap v125: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-22T06:41:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T06:41:20.096 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 48, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 70, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 73, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1832, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1835, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1371, in rank_tell return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/cephfs/filesystem.py", line 1358, in mds_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi105 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls' 2024-04-22T06:41:20.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:20 smithi105 ceph-mon[27916]: pgmap v126: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:41:20.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:20 smithi112 ceph-mon[31236]: pgmap v126: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:41:20.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:20 smithi177 ceph-mon[31346]: pgmap v126: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:41:22.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:22 smithi105 ceph-mon[27916]: pgmap v127: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:41:22.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:22 smithi112 ceph-mon[31236]: pgmap v127: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:41:22.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:22 smithi177 ceph-mon[31346]: pgmap v127: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:41:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:24 smithi105 ceph-mon[27916]: pgmap v128: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:41:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:24 smithi112 ceph-mon[31236]: pgmap v128: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:41:24.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:24 smithi177 ceph-mon[31346]: pgmap v128: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:41:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:26 smithi105 ceph-mon[27916]: pgmap v129: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:26 smithi112 ceph-mon[31236]: pgmap v129: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:26 smithi177 ceph-mon[31346]: pgmap v129: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:28 smithi105 ceph-mon[27916]: pgmap v130: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:28 smithi112 ceph-mon[31236]: pgmap v130: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:28.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:28 smithi177 ceph-mon[31346]: pgmap v130: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:30.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:30 smithi105 ceph-mon[27916]: pgmap v131: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:30 smithi112 ceph-mon[31236]: pgmap v131: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:30 smithi177 ceph-mon[31346]: pgmap v131: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:32 smithi105 ceph-mon[27916]: pgmap v132: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:32.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:32 smithi112 ceph-mon[31236]: pgmap v132: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:32.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:32 smithi177 ceph-mon[31346]: pgmap v132: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:34.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:34 smithi105 ceph-mon[27916]: pgmap v133: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:34.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:34 smithi112 ceph-mon[31236]: pgmap v133: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:34.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:34 smithi177 ceph-mon[31346]: pgmap v133: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:36.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:36 smithi105 ceph-mon[27916]: pgmap v134: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:36.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:36 smithi112 ceph-mon[31236]: pgmap v134: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:36.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:36 smithi177 ceph-mon[31346]: pgmap v134: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:38 smithi105 ceph-mon[27916]: pgmap v135: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:38 smithi112 ceph-mon[31236]: pgmap v135: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:38.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:38 smithi177 ceph-mon[31346]: pgmap v135: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:40.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:40 smithi105 ceph-mon[27916]: pgmap v136: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:40.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:40 smithi112 ceph-mon[31236]: pgmap v136: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:40.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:40 smithi177 ceph-mon[31346]: pgmap v136: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:42.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:42 smithi112 ceph-mon[31236]: pgmap v137: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:42.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:42 smithi177 ceph-mon[31346]: pgmap v137: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:42.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:42 smithi105 ceph-mon[27916]: pgmap v137: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:44.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:44 smithi112 ceph-mon[31236]: pgmap v138: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:44.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:44 smithi177 ceph-mon[31346]: pgmap v138: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:44.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:44 smithi105 ceph-mon[27916]: pgmap v138: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:46 smithi112 ceph-mon[31236]: pgmap v139: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:46.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:41:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:46 smithi177 ceph-mon[31346]: pgmap v139: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:46.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:41:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:46 smithi105 ceph-mon[27916]: pgmap v139: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:41:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:41:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:41:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:41:48.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:48 smithi112 ceph-mon[31236]: pgmap v140: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:48.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:48 smithi177 ceph-mon[31346]: pgmap v140: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:48 smithi105 ceph-mon[27916]: pgmap v140: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:50 smithi112 ceph-mon[31236]: pgmap v141: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:50.470 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:50 smithi105 ceph-mon[27916]: pgmap v141: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:50 smithi177 ceph-mon[31346]: pgmap v141: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:52 smithi112 ceph-mon[31236]: pgmap v142: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:52.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:52 smithi177 ceph-mon[31346]: pgmap v142: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:52.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:52 smithi105 ceph-mon[27916]: pgmap v142: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:54 smithi112 ceph-mon[31236]: pgmap v143: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:54 smithi177 ceph-mon[31346]: pgmap v143: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:54.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:54 smithi105 ceph-mon[27916]: pgmap v143: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:41:56.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:56 smithi112 ceph-mon[31236]: pgmap v144: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:56.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:56 smithi177 ceph-mon[31346]: pgmap v144: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:56.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:56 smithi105 ceph-mon[27916]: pgmap v144: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:58.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:41:58 smithi112 ceph-mon[31236]: pgmap v145: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:58.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:41:58 smithi177 ceph-mon[31346]: pgmap v145: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:41:58.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:41:58 smithi105 ceph-mon[27916]: pgmap v145: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:00.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:00 smithi112 ceph-mon[31236]: pgmap v146: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:00.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:00 smithi177 ceph-mon[31346]: pgmap v146: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:00.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:00 smithi105 ceph-mon[27916]: pgmap v146: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:02 smithi112 ceph-mon[31236]: pgmap v147: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:02 smithi177 ceph-mon[31346]: pgmap v147: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:02.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:02 smithi105 ceph-mon[27916]: pgmap v147: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:04 smithi112 ceph-mon[31236]: pgmap v148: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:04.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:04 smithi177 ceph-mon[31346]: pgmap v148: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:04 smithi105 ceph-mon[27916]: pgmap v148: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:06.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:06 smithi112 ceph-mon[31236]: pgmap v149: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:06.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:06 smithi177 ceph-mon[31346]: pgmap v149: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:06.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:06 smithi105 ceph-mon[27916]: pgmap v149: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:08.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:08 smithi112 ceph-mon[31236]: pgmap v150: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:08 smithi177 ceph-mon[31346]: pgmap v150: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:08.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:08 smithi105 ceph-mon[27916]: pgmap v150: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:10.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:10 smithi177 ceph-mon[31346]: pgmap v151: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:10.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:10 smithi105 ceph-mon[27916]: pgmap v151: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:10.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:10 smithi112 ceph-mon[31236]: pgmap v151: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:12 smithi177 ceph-mon[31346]: pgmap v152: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:12.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:12 smithi112 ceph-mon[31236]: pgmap v152: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:12.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:12 smithi105 ceph-mon[27916]: pgmap v152: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:42:14.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:14 smithi177 ceph-mon[31346]: pgmap v153: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:14.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:14 smithi105 ceph-mon[27916]: pgmap v153: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:14.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:14 smithi112 ceph-mon[31236]: pgmap v153: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:42:16.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:16 smithi177 ceph-mon[31346]: pgmap v154: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:16.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:16 smithi105 ceph-mon[27916]: pgmap v154: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:16.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:16 smithi112 ceph-mon[31236]: pgmap v154: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:18.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:18 smithi177 ceph-mon[31346]: pgmap v155: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:18.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:18 smithi105 ceph-mon[27916]: pgmap v155: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:18.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:18 smithi112 ceph-mon[31236]: pgmap v155: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:42:20.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:20 smithi177 ceph-mon[31346]: pgmap v156: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:42:20.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:20 smithi105 ceph-mon[27916]: pgmap v156: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:42:20.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:20 smithi112 ceph-mon[31236]: pgmap v156: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:42:22.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:22 smithi177 ceph-mon[31346]: pgmap v157: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:22.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:22 smithi105 ceph-mon[27916]: pgmap v157: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:22.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:22 smithi112 ceph-mon[31236]: pgmap v157: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:24.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:24 smithi177 ceph-mon[31346]: pgmap v158: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:24.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:24 smithi105 ceph-mon[27916]: pgmap v158: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:24.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:24 smithi112 ceph-mon[31236]: pgmap v158: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:26 smithi177 ceph-mon[31346]: pgmap v159: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:26 smithi105 ceph-mon[27916]: pgmap v159: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:26.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:26 smithi112 ceph-mon[31236]: pgmap v159: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:28.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:28 smithi177 ceph-mon[31346]: pgmap v160: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:28 smithi105 ceph-mon[27916]: pgmap v160: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:28.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:28 smithi112 ceph-mon[31236]: pgmap v160: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:30 smithi177 ceph-mon[31346]: pgmap v161: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:30.510 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:30 smithi112 ceph-mon[31236]: pgmap v161: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:30.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:30 smithi105 ceph-mon[27916]: pgmap v161: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:32.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:32 smithi105 ceph-mon[27916]: pgmap v162: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:32.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:32 smithi112 ceph-mon[31236]: pgmap v162: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:32.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:32 smithi177 ceph-mon[31346]: pgmap v162: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:34 smithi105 ceph-mon[27916]: pgmap v163: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:34.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:34 smithi112 ceph-mon[31236]: pgmap v163: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:34.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:34 smithi177 ceph-mon[31346]: pgmap v163: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:36.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:36 smithi105 ceph-mon[27916]: pgmap v164: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:36.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:36 smithi112 ceph-mon[31236]: pgmap v164: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:36.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:36 smithi177 ceph-mon[31346]: pgmap v164: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:38.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:38 smithi105 ceph-mon[27916]: pgmap v165: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:38.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:38 smithi112 ceph-mon[31236]: pgmap v165: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:38.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:38 smithi177 ceph-mon[31346]: pgmap v165: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:40 smithi105 ceph-mon[27916]: pgmap v166: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:40.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:40 smithi112 ceph-mon[31236]: pgmap v166: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:40.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:40 smithi177 ceph-mon[31346]: pgmap v166: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:42.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:42 smithi105 ceph-mon[27916]: pgmap v167: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:42.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:42 smithi112 ceph-mon[31236]: pgmap v167: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:42.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:42 smithi177 ceph-mon[31346]: pgmap v167: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:44.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:44 smithi105 ceph-mon[27916]: pgmap v168: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:44.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:44 smithi112 ceph-mon[31236]: pgmap v168: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:44.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:44 smithi177 ceph-mon[31346]: pgmap v168: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:46 smithi105 ceph-mon[27916]: pgmap v169: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:46 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:46.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:46 smithi112 ceph-mon[31236]: pgmap v169: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:46.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:46 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:46.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:46 smithi177 ceph-mon[31346]: pgmap v169: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:46.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:46 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:42:47.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:47.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:47.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:42:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:42:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:42:47.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:42:47.740 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:42:48.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:48 smithi177 ceph-mon[31346]: pgmap v170: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:48 smithi105 ceph-mon[27916]: pgmap v170: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:48 smithi112 ceph-mon[31236]: pgmap v170: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:50.970 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:50 smithi105 ceph-mon[27916]: pgmap v171: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:50.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:50 smithi177 ceph-mon[31346]: pgmap v171: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:51.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:50 smithi112 ceph-mon[31236]: pgmap v171: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:52 smithi177 ceph-mon[31346]: pgmap v172: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:53.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:52 smithi105 ceph-mon[27916]: pgmap v172: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:53.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:52 smithi112 ceph-mon[31236]: pgmap v172: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:54.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:54 smithi177 ceph-mon[31346]: pgmap v173: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:55.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:54 smithi105 ceph-mon[27916]: pgmap v173: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:55.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:54 smithi112 ceph-mon[31236]: pgmap v173: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:42:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:56 smithi177 ceph-mon[31346]: pgmap v174: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:57.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:56 smithi105 ceph-mon[27916]: pgmap v174: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:57.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:56 smithi112 ceph-mon[31236]: pgmap v174: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:58.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:42:58 smithi177 ceph-mon[31346]: pgmap v175: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:59.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:42:58 smithi105 ceph-mon[27916]: pgmap v175: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:42:59.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:42:58 smithi112 ceph-mon[31236]: pgmap v175: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:00.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:00 smithi177 ceph-mon[31346]: pgmap v176: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:01.010 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:00 smithi112 ceph-mon[31236]: pgmap v176: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:01.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:00 smithi105 ceph-mon[27916]: pgmap v176: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:02.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:02 smithi177 ceph-mon[31346]: pgmap v177: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:03.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:02 smithi105 ceph-mon[27916]: pgmap v177: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:03.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:02 smithi112 ceph-mon[31236]: pgmap v177: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:04.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:04 smithi177 ceph-mon[31346]: pgmap v178: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:05.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:04 smithi105 ceph-mon[27916]: pgmap v178: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:05.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:04 smithi112 ceph-mon[31236]: pgmap v178: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:06.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:06 smithi177 ceph-mon[31346]: pgmap v179: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:07.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:06 smithi105 ceph-mon[27916]: pgmap v179: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:07.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:06 smithi112 ceph-mon[31236]: pgmap v179: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:09.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:08 smithi105 ceph-mon[27916]: pgmap v180: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:09.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:08 smithi112 ceph-mon[31236]: pgmap v180: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:09.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:08 smithi177 ceph-mon[31346]: pgmap v180: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:11.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:10 smithi105 ceph-mon[27916]: pgmap v181: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:11.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:10 smithi112 ceph-mon[31236]: pgmap v181: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:11.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:10 smithi177 ceph-mon[31346]: pgmap v181: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:13.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:12 smithi105 ceph-mon[27916]: pgmap v182: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:13.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:12 smithi112 ceph-mon[31236]: pgmap v182: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:13.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:12 smithi177 ceph-mon[31346]: pgmap v182: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-22T06:43:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:14 smithi105 ceph-mon[27916]: pgmap v183: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:15.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:14 smithi112 ceph-mon[31236]: pgmap v183: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:15.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:14 smithi177 ceph-mon[31346]: pgmap v183: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-22T06:43:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:16 smithi105 ceph-mon[27916]: pgmap v184: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:17.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:16 smithi112 ceph-mon[31236]: pgmap v184: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:17.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:16 smithi177 ceph-mon[31346]: pgmap v184: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:19.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:18 smithi105 ceph-mon[27916]: pgmap v185: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:19.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:18 smithi112 ceph-mon[31236]: pgmap v185: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:19.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:18 smithi177 ceph-mon[31346]: pgmap v185: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-22T06:43:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:20 smithi105 ceph-mon[27916]: pgmap v186: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:43:21.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:20 smithi112 ceph-mon[31236]: pgmap v186: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:43:21.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:20 smithi177 ceph-mon[31346]: pgmap v186: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:43:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:22 smithi105 ceph-mon[27916]: pgmap v187: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:23.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:22 smithi112 ceph-mon[31236]: pgmap v187: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:23.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:22 smithi177 ceph-mon[31346]: pgmap v187: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:24 smithi105 ceph-mon[27916]: pgmap v188: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:25.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:24 smithi112 ceph-mon[31236]: pgmap v188: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:25.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:24 smithi177 ceph-mon[31346]: pgmap v188: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:27.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:26 smithi105 ceph-mon[27916]: pgmap v189: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:27.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:26 smithi112 ceph-mon[31236]: pgmap v189: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:27.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:26 smithi177 ceph-mon[31346]: pgmap v189: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:29.041 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:28 smithi177 ceph-mon[31346]: pgmap v190: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:28 smithi105 ceph-mon[27916]: pgmap v190: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:29.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:28 smithi112 ceph-mon[31236]: pgmap v190: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:30 smithi105 ceph-mon[27916]: pgmap v191: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:31.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:30 smithi112 ceph-mon[31236]: pgmap v191: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:31.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:30 smithi177 ceph-mon[31346]: pgmap v191: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:33.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:32 smithi105 ceph-mon[27916]: pgmap v192: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:33.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:32 smithi112 ceph-mon[31236]: pgmap v192: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:33.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:32 smithi177 ceph-mon[31346]: pgmap v192: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:35.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:34 smithi105 ceph-mon[27916]: pgmap v193: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:35.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:34 smithi112 ceph-mon[31236]: pgmap v193: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:35.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:34 smithi177 ceph-mon[31346]: pgmap v193: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:36 smithi105 ceph-mon[27916]: pgmap v194: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:36 smithi112 ceph-mon[31236]: pgmap v194: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:37.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:36 smithi177 ceph-mon[31346]: pgmap v194: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:39.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:38 smithi105 ceph-mon[27916]: pgmap v195: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:39.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:38 smithi112 ceph-mon[31236]: pgmap v195: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:39.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:38 smithi177 ceph-mon[31346]: pgmap v195: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:41.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:40 smithi105 ceph-mon[27916]: pgmap v196: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:41.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:40 smithi112 ceph-mon[31236]: pgmap v196: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:41.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:40 smithi177 ceph-mon[31346]: pgmap v196: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:42 smithi105 ceph-mon[27916]: pgmap v197: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:42 smithi112 ceph-mon[31236]: pgmap v197: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:43.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:42 smithi177 ceph-mon[31346]: pgmap v197: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:44 smithi105 ceph-mon[27916]: pgmap v198: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:44 smithi112 ceph-mon[31236]: pgmap v198: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:45.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:44 smithi177 ceph-mon[31346]: pgmap v198: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:47.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:46 smithi105 ceph-mon[27916]: pgmap v199: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:47.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:46 smithi112 ceph-mon[31236]: pgmap v199: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:47.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:46 smithi177 ceph-mon[31346]: pgmap v199: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:43:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:48.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:43:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:43:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:43:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:43:48.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:43:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:48 smithi105 ceph-mon[27916]: pgmap v200: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:48 smithi112 ceph-mon[31236]: pgmap v200: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:48 smithi177 ceph-mon[31346]: pgmap v200: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:51.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:50 smithi105 ceph-mon[27916]: pgmap v201: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:51.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:50 smithi112 ceph-mon[31236]: pgmap v201: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:51.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:50 smithi177 ceph-mon[31346]: pgmap v201: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:53.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:52 smithi105 ceph-mon[27916]: pgmap v202: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:53.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:52 smithi112 ceph-mon[31236]: pgmap v202: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:53.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:52 smithi177 ceph-mon[31346]: pgmap v202: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:55.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:54 smithi105 ceph-mon[27916]: pgmap v203: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:55.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:54 smithi112 ceph-mon[31236]: pgmap v203: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:55.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:54 smithi177 ceph-mon[31346]: pgmap v203: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:43:57.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:56 smithi105 ceph-mon[27916]: pgmap v204: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:57.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:56 smithi112 ceph-mon[31236]: pgmap v204: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:57.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:56 smithi177 ceph-mon[31346]: pgmap v204: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:59.170 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:43:58 smithi105 ceph-mon[27916]: pgmap v205: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:59.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:43:58 smithi112 ceph-mon[31236]: pgmap v205: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:43:59.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:43:58 smithi177 ceph-mon[31346]: pgmap v205: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:01.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:00 smithi105 ceph-mon[27916]: pgmap v206: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:01.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:00 smithi112 ceph-mon[31236]: pgmap v206: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:01.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:00 smithi177 ceph-mon[31346]: pgmap v206: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:03.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:02 smithi105 ceph-mon[27916]: pgmap v207: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:03.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:02 smithi112 ceph-mon[31236]: pgmap v207: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:03.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:02 smithi177 ceph-mon[31346]: pgmap v207: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:05.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:04 smithi105 ceph-mon[27916]: pgmap v208: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:44:05.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:04 smithi112 ceph-mon[31236]: pgmap v208: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:44:05.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:04 smithi177 ceph-mon[31346]: pgmap v208: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:44:07.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:06 smithi105 ceph-mon[27916]: pgmap v209: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:07.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:06 smithi112 ceph-mon[31236]: pgmap v209: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:07.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:06 smithi177 ceph-mon[31346]: pgmap v209: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:09.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:08 smithi105 ceph-mon[27916]: pgmap v210: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:09.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:08 smithi112 ceph-mon[31236]: pgmap v210: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:09.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:08 smithi177 ceph-mon[31346]: pgmap v210: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:44:11.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:10 smithi105 ceph-mon[27916]: pgmap v211: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-22T06:44:11.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:10 smithi112 ceph-mon[31236]: pgmap v211: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-22T06:44:11.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:10 smithi177 ceph-mon[31346]: pgmap v211: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-22T06:44:12.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:11 smithi105 ceph-mon[27916]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:44:12.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:11 smithi112 ceph-mon[31236]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:44:12.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:11 smithi177 ceph-mon[31346]: osdmap e105: 12 total, 12 up, 12 in 2024-04-22T06:44:13.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:12 smithi105 ceph-mon[27916]: pgmap v213: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:44:13.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:12 smithi112 ceph-mon[31236]: pgmap v213: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:44:13.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:12 smithi177 ceph-mon[31346]: pgmap v213: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:44:15.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:14 smithi105 ceph-mon[27916]: pgmap v214: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:15.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:14 smithi112 ceph-mon[31236]: pgmap v214: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:15.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:14 smithi177 ceph-mon[31346]: pgmap v214: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:16 smithi105 ceph-mon[27916]: pgmap v215: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:17.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:16 smithi105 ceph-mon[27916]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:44:17.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:16 smithi112 ceph-mon[31236]: pgmap v215: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:17.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:16 smithi112 ceph-mon[31236]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:44:17.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:16 smithi177 ceph-mon[31346]: pgmap v215: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:17.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:16 smithi177 ceph-mon[31346]: osdmap e106: 12 total, 12 up, 12 in 2024-04-22T06:44:19.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:18 smithi105 ceph-mon[27916]: pgmap v217: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:44:19.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:18 smithi112 ceph-mon[31236]: pgmap v217: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:44:19.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:18 smithi177 ceph-mon[31346]: pgmap v217: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:44:21.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:20 smithi105 ceph-mon[27916]: pgmap v218: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 374 B/s wr, 12 op/s 2024-04-22T06:44:21.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:20 smithi112 ceph-mon[31236]: pgmap v218: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 374 B/s wr, 12 op/s 2024-04-22T06:44:21.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:20 smithi177 ceph-mon[31346]: pgmap v218: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 374 B/s wr, 12 op/s 2024-04-22T06:44:23.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:22 smithi105 ceph-mon[27916]: pgmap v219: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T06:44:23.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:22 smithi112 ceph-mon[31236]: pgmap v219: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T06:44:23.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:22 smithi177 ceph-mon[31346]: pgmap v219: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-04-22T06:44:25.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:24 smithi105 ceph-mon[27916]: pgmap v220: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:25.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:24 smithi112 ceph-mon[31236]: pgmap v220: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:25.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:24 smithi177 ceph-mon[31346]: pgmap v220: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:27.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:26 smithi105 ceph-mon[27916]: pgmap v221: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:27.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:26 smithi112 ceph-mon[31236]: pgmap v221: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:27.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:26 smithi177 ceph-mon[31346]: pgmap v221: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:44:29.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:28 smithi105 ceph-mon[27916]: pgmap v222: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 187 B/s wr, 9 op/s 2024-04-22T06:44:29.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:28 smithi112 ceph-mon[31236]: pgmap v222: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 187 B/s wr, 9 op/s 2024-04-22T06:44:29.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:28 smithi177 ceph-mon[31346]: pgmap v222: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 187 B/s wr, 9 op/s 2024-04-22T06:44:31.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:30 smithi105 ceph-mon[27916]: pgmap v223: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:44:31.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:30 smithi112 ceph-mon[31236]: pgmap v223: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:44:31.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:30 smithi177 ceph-mon[31346]: pgmap v223: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-22T06:44:33.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:32 smithi105 ceph-mon[27916]: pgmap v224: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:33.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:32 smithi112 ceph-mon[31236]: pgmap v224: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:33.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:32 smithi177 ceph-mon[31346]: pgmap v224: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:35.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:34 smithi105 ceph-mon[27916]: pgmap v225: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:35.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:34 smithi112 ceph-mon[31236]: pgmap v225: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:35.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:34 smithi177 ceph-mon[31346]: pgmap v225: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:37.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:36 smithi105 ceph-mon[27916]: pgmap v226: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:37.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:36 smithi112 ceph-mon[31236]: pgmap v226: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:37.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:36 smithi177 ceph-mon[31346]: pgmap v226: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:39.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:38 smithi105 ceph-mon[27916]: pgmap v227: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:39.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:38 smithi112 ceph-mon[31236]: pgmap v227: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:39.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:38 smithi177 ceph-mon[31346]: pgmap v227: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:41.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:40 smithi105 ceph-mon[27916]: pgmap v228: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:41.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:40 smithi112 ceph-mon[31236]: pgmap v228: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:41.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:40 smithi177 ceph-mon[31346]: pgmap v228: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:43.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:42 smithi105 ceph-mon[27916]: pgmap v229: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:43.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:42 smithi112 ceph-mon[31236]: pgmap v229: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:43.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:42 smithi177 ceph-mon[31346]: pgmap v229: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:45.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:44 smithi105 ceph-mon[27916]: pgmap v230: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:45.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:44 smithi112 ceph-mon[31236]: pgmap v230: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:45.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:44 smithi177 ceph-mon[31346]: pgmap v230: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:47.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:46 smithi105 ceph-mon[27916]: pgmap v231: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:47.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:46 smithi112 ceph-mon[31236]: pgmap v231: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:47.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:46 smithi177 ceph-mon[31346]: pgmap v231: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:48.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:47 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:48.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:47 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:48.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:47 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:44:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:48 smithi105 ceph-mon[27916]: pgmap v232: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:48 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:48 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:49.168 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:48 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:44:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:48 smithi112 ceph-mon[31236]: pgmap v232: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:48 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:49.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:48 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:49.209 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:48 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:44:49.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:48 smithi177 ceph-mon[31346]: pgmap v232: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:49.239 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:48 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:44:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:48 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:44:49.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:48 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:44:50.208 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:49 smithi112 ceph-mon[31236]: pgmap v233: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:50.240 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:49 smithi177 ceph-mon[31346]: pgmap v233: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:49 smithi105 ceph-mon[27916]: pgmap v233: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:52.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:52 smithi105 ceph-mon[27916]: pgmap v234: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:52 smithi112 ceph-mon[31236]: pgmap v234: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:52.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:52 smithi177 ceph-mon[31346]: pgmap v234: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:54.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:54 smithi105 ceph-mon[27916]: pgmap v235: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:54 smithi112 ceph-mon[31236]: pgmap v235: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:54 smithi177 ceph-mon[31346]: pgmap v235: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:44:56.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:56 smithi105 ceph-mon[27916]: pgmap v236: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:56.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:56 smithi112 ceph-mon[31236]: pgmap v236: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:56.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:56 smithi177 ceph-mon[31346]: pgmap v236: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:58.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:44:58 smithi105 ceph-mon[27916]: pgmap v237: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:58.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:44:58 smithi112 ceph-mon[31236]: pgmap v237: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:44:58.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:44:58 smithi177 ceph-mon[31346]: pgmap v237: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:00.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:00 smithi105 ceph-mon[27916]: pgmap v238: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:00.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:00 smithi112 ceph-mon[31236]: pgmap v238: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:00.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:00 smithi177 ceph-mon[31346]: pgmap v238: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:02.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:02 smithi105 ceph-mon[27916]: pgmap v239: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:02.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:02 smithi112 ceph-mon[31236]: pgmap v239: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:02 smithi177 ceph-mon[31346]: pgmap v239: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:04.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:04 smithi105 ceph-mon[27916]: pgmap v240: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:04.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:04 smithi112 ceph-mon[31236]: pgmap v240: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:04.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:04 smithi177 ceph-mon[31346]: pgmap v240: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:06.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:06 smithi105 ceph-mon[27916]: pgmap v241: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:06.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:06 smithi112 ceph-mon[31236]: pgmap v241: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:06.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:06 smithi177 ceph-mon[31346]: pgmap v241: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:08.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:08 smithi105 ceph-mon[27916]: pgmap v242: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:08.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:08 smithi112 ceph-mon[31236]: pgmap v242: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:08 smithi177 ceph-mon[31346]: pgmap v242: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:10.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:10 smithi105 ceph-mon[27916]: pgmap v243: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:10.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:10 smithi112 ceph-mon[31236]: pgmap v243: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:10.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:10 smithi177 ceph-mon[31346]: pgmap v243: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:45:12.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:12 smithi105 ceph-mon[27916]: pgmap v244: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:12.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:12 smithi105 ceph-mon[27916]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:45:12.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:12 smithi112 ceph-mon[31236]: pgmap v244: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:12.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:12 smithi112 ceph-mon[31236]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:45:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:12 smithi177 ceph-mon[31346]: pgmap v244: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:45:12.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:12 smithi177 ceph-mon[31346]: osdmap e107: 12 total, 12 up, 12 in 2024-04-22T06:45:14.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:14 smithi105 ceph-mon[27916]: pgmap v246: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:14.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:14 smithi112 ceph-mon[31236]: pgmap v246: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:14.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:14 smithi177 ceph-mon[31346]: pgmap v246: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:16.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:16 smithi105 ceph-mon[27916]: pgmap v247: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:16.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:16 smithi105 ceph-mon[27916]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:45:16.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:16 smithi112 ceph-mon[31236]: pgmap v247: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:16.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:16 smithi112 ceph-mon[31236]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:45:16.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:16 smithi177 ceph-mon[31346]: pgmap v247: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:45:16.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:16 smithi177 ceph-mon[31346]: osdmap e108: 12 total, 12 up, 12 in 2024-04-22T06:45:18.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:18 smithi105 ceph-mon[27916]: pgmap v249: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:45:18.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:18 smithi112 ceph-mon[31236]: pgmap v249: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:45:18.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:18 smithi177 ceph-mon[31346]: pgmap v249: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:45:20.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:20 smithi105 ceph-mon[27916]: pgmap v250: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:45:20.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:20 smithi112 ceph-mon[31236]: pgmap v250: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:45:20.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:20 smithi177 ceph-mon[31346]: pgmap v250: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:45:22.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:22 smithi105 ceph-mon[27916]: pgmap v251: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-04-22T06:45:22.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:22 smithi112 ceph-mon[31236]: pgmap v251: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-04-22T06:45:22.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:22 smithi177 ceph-mon[31346]: pgmap v251: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2024-04-22T06:45:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:24 smithi105 ceph-mon[27916]: pgmap v252: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:24 smithi112 ceph-mon[31236]: pgmap v252: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:24.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:24 smithi177 ceph-mon[31346]: pgmap v252: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:26 smithi105 ceph-mon[27916]: pgmap v253: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:26 smithi112 ceph-mon[31236]: pgmap v253: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:26.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:26 smithi177 ceph-mon[31346]: pgmap v253: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:45:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:28 smithi105 ceph-mon[27916]: pgmap v254: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:45:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:28 smithi112 ceph-mon[31236]: pgmap v254: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:45:28.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:28 smithi177 ceph-mon[31346]: pgmap v254: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:45:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:30 smithi112 ceph-mon[31236]: pgmap v255: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:45:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:30 smithi177 ceph-mon[31346]: pgmap v255: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:45:30.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:30 smithi105 ceph-mon[27916]: pgmap v255: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:45:32.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:32 smithi112 ceph-mon[31236]: pgmap v256: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:32.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:32 smithi177 ceph-mon[31346]: pgmap v256: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:32.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:32 smithi105 ceph-mon[27916]: pgmap v256: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:34.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:34 smithi112 ceph-mon[31236]: pgmap v257: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:34.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:34 smithi177 ceph-mon[31346]: pgmap v257: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:34 smithi105 ceph-mon[27916]: pgmap v257: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:36.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:36 smithi112 ceph-mon[31236]: pgmap v258: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:36.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:36 smithi177 ceph-mon[31346]: pgmap v258: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:36.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:36 smithi105 ceph-mon[27916]: pgmap v258: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:38 smithi112 ceph-mon[31236]: pgmap v259: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:38.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:38 smithi177 ceph-mon[31346]: pgmap v259: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:38.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:38 smithi105 ceph-mon[27916]: pgmap v259: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-22T06:45:40.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:40 smithi112 ceph-mon[31236]: pgmap v260: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:40.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:40 smithi177 ceph-mon[31346]: pgmap v260: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:40 smithi105 ceph-mon[27916]: pgmap v260: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:42.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:42 smithi112 ceph-mon[31236]: pgmap v261: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:42.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:42 smithi177 ceph-mon[31346]: pgmap v261: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:42.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:42 smithi105 ceph-mon[27916]: pgmap v261: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:44.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:44 smithi112 ceph-mon[31236]: pgmap v262: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:44.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:44 smithi177 ceph-mon[31346]: pgmap v262: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:44.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:44 smithi105 ceph-mon[27916]: pgmap v262: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:46 smithi112 ceph-mon[31236]: pgmap v263: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:46.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:46 smithi177 ceph-mon[31346]: pgmap v263: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:46 smithi105 ceph-mon[27916]: pgmap v263: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:48.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:48 smithi112 ceph-mon[31236]: pgmap v264: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:48.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:48 smithi177 ceph-mon[31346]: pgmap v264: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:48.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:48 smithi105 ceph-mon[27916]: pgmap v264: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:49 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:49 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:49 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:49.459 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:49 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:45:49.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:49 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:49.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:49 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:49 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:49.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:49 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:45:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:49 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:45:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:49 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:45:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:49 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:45:49.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:49 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:45:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:50 smithi112 ceph-mon[31236]: pgmap v265: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:50 smithi177 ceph-mon[31346]: pgmap v265: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:50.569 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:50 smithi105 ceph-mon[27916]: pgmap v265: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:52 smithi112 ceph-mon[31236]: pgmap v266: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:52.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:52 smithi177 ceph-mon[31346]: pgmap v266: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:52.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:52 smithi105 ceph-mon[27916]: pgmap v266: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:54 smithi112 ceph-mon[31236]: pgmap v267: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:54 smithi177 ceph-mon[31346]: pgmap v267: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:54.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:54 smithi105 ceph-mon[27916]: pgmap v267: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:45:56.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:56 smithi177 ceph-mon[31346]: pgmap v268: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:56.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:56 smithi105 ceph-mon[27916]: pgmap v268: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:56.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:56 smithi112 ceph-mon[31236]: pgmap v268: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:58.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:45:58 smithi177 ceph-mon[31346]: pgmap v269: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:58.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:45:58 smithi105 ceph-mon[27916]: pgmap v269: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:45:58.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:45:58 smithi112 ceph-mon[31236]: pgmap v269: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:00.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:00 smithi177 ceph-mon[31346]: pgmap v270: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:00.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:00 smithi105 ceph-mon[27916]: pgmap v270: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:00.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:00 smithi112 ceph-mon[31236]: pgmap v270: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:02 smithi177 ceph-mon[31346]: pgmap v271: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:02.510 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:02 smithi112 ceph-mon[31236]: pgmap v271: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:02.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:02 smithi105 ceph-mon[27916]: pgmap v271: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:04.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:04 smithi177 ceph-mon[31346]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:04 smithi105 ceph-mon[27916]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:04 smithi112 ceph-mon[31236]: pgmap v272: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:06.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:06 smithi177 ceph-mon[31346]: pgmap v273: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:06.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:06 smithi105 ceph-mon[27916]: pgmap v273: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:06.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:06 smithi112 ceph-mon[31236]: pgmap v273: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:08 smithi177 ceph-mon[31346]: pgmap v274: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:08.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:08 smithi105 ceph-mon[27916]: pgmap v274: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:08.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:08 smithi112 ceph-mon[31236]: pgmap v274: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:10.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:10 smithi177 ceph-mon[31346]: pgmap v275: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:10.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:10 smithi105 ceph-mon[27916]: pgmap v275: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:10.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:10 smithi112 ceph-mon[31236]: pgmap v275: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:46:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:12 smithi177 ceph-mon[31346]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:12 smithi177 ceph-mon[31346]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:46:12.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:12 smithi105 ceph-mon[27916]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:12.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:12 smithi105 ceph-mon[27916]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:46:12.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:12 smithi112 ceph-mon[31236]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:46:12.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:12 smithi112 ceph-mon[31236]: osdmap e109: 12 total, 12 up, 12 in 2024-04-22T06:46:14.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:14 smithi177 ceph-mon[31346]: pgmap v278: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:14.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:14 smithi105 ceph-mon[27916]: pgmap v278: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:14.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:14 smithi112 ceph-mon[31236]: pgmap v278: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:16.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:16 smithi105 ceph-mon[27916]: pgmap v279: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:16.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:16 smithi112 ceph-mon[31236]: pgmap v279: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:16.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:16 smithi177 ceph-mon[31346]: pgmap v279: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:18.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:18 smithi105 ceph-mon[27916]: pgmap v280: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:18.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:18 smithi112 ceph-mon[31236]: pgmap v280: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:18.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:18 smithi177 ceph-mon[31346]: pgmap v280: 97 pgs: 15 active+clean+snaptrim_wait, 73 active+clean, 9 active+clean+snaptrim; 70 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:20.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:20 smithi105 ceph-mon[27916]: pgmap v281: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:20.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:20 smithi112 ceph-mon[31236]: pgmap v281: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:20.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:20 smithi177 ceph-mon[31346]: pgmap v281: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:22.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:22 smithi105 ceph-mon[27916]: pgmap v282: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:22.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:22 smithi105 ceph-mon[27916]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:46:22.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:22 smithi112 ceph-mon[31236]: pgmap v282: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:22.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:22 smithi112 ceph-mon[31236]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:46:22.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:22 smithi177 ceph-mon[31346]: pgmap v282: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:46:22.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:22 smithi177 ceph-mon[31346]: osdmap e110: 12 total, 12 up, 12 in 2024-04-22T06:46:24.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:24 smithi105 ceph-mon[27916]: pgmap v284: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:24.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:24 smithi112 ceph-mon[31236]: pgmap v284: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:24.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:24 smithi177 ceph-mon[31346]: pgmap v284: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:26.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:26 smithi105 ceph-mon[27916]: pgmap v285: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:26.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:26 smithi112 ceph-mon[31236]: pgmap v285: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:26.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:26 smithi177 ceph-mon[31346]: pgmap v285: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:28 smithi105 ceph-mon[27916]: pgmap v286: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:28 smithi112 ceph-mon[31236]: pgmap v286: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:28.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:28 smithi177 ceph-mon[31346]: pgmap v286: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:46:30.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:30 smithi105 ceph-mon[27916]: pgmap v287: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:30.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:30 smithi112 ceph-mon[31236]: pgmap v287: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:30.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:30 smithi177 ceph-mon[31346]: pgmap v287: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:32.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:32 smithi105 ceph-mon[27916]: pgmap v288: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:32.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:32 smithi112 ceph-mon[31236]: pgmap v288: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:32.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:32 smithi177 ceph-mon[31346]: pgmap v288: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-22T06:46:34.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:34 smithi105 ceph-mon[27916]: pgmap v289: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-04-22T06:46:34.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:34 smithi112 ceph-mon[31236]: pgmap v289: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-04-22T06:46:34.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:34 smithi177 ceph-mon[31346]: pgmap v289: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-04-22T06:46:36.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:36 smithi105 ceph-mon[27916]: pgmap v290: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:36.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:36 smithi112 ceph-mon[31236]: pgmap v290: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:36.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:36 smithi177 ceph-mon[31346]: pgmap v290: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:38.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:38 smithi105 ceph-mon[27916]: pgmap v291: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:38.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:38 smithi112 ceph-mon[31236]: pgmap v291: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:38.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:38 smithi177 ceph-mon[31346]: pgmap v291: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:40.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:40 smithi105 ceph-mon[27916]: pgmap v292: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:40.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:40 smithi112 ceph-mon[31236]: pgmap v292: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:40.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:40 smithi177 ceph-mon[31346]: pgmap v292: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:42.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:42 smithi112 ceph-mon[31236]: pgmap v293: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:42.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:42 smithi177 ceph-mon[31346]: pgmap v293: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:42.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:42 smithi105 ceph-mon[27916]: pgmap v293: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:44.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:44 smithi112 ceph-mon[31236]: pgmap v294: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:44.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:44 smithi177 ceph-mon[31346]: pgmap v294: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:44.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:44 smithi105 ceph-mon[27916]: pgmap v294: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:46.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:46 smithi112 ceph-mon[31236]: pgmap v295: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:46.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:46 smithi177 ceph-mon[31346]: pgmap v295: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:46.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:46 smithi105 ceph-mon[27916]: pgmap v295: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:48.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:48 smithi112 ceph-mon[31236]: pgmap v296: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:48.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:48 smithi177 ceph-mon[31346]: pgmap v296: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:48.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:48 smithi105 ceph-mon[27916]: pgmap v296: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:49.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:49 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:49.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:49 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:49.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:49 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:46:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:50 smithi112 ceph-mon[31236]: pgmap v297: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:50 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:50 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:50.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:50 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:46:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:50 smithi177 ceph-mon[31346]: pgmap v297: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:50 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:50 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:50.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:50 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:46:50.569 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:50 smithi105 ceph-mon[27916]: pgmap v297: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:50.569 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:50 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:46:50.569 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:50 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:46:50.569 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:50 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:46:52.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:52 smithi112 ceph-mon[31236]: pgmap v298: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:52.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:52 smithi177 ceph-mon[31346]: pgmap v298: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:52.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:52 smithi105 ceph-mon[27916]: pgmap v298: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:54.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:54 smithi112 ceph-mon[31236]: pgmap v299: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:54.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:54 smithi177 ceph-mon[31346]: pgmap v299: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:54.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:54 smithi105 ceph-mon[27916]: pgmap v299: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:46:56.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:56 smithi112 ceph-mon[31236]: pgmap v300: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:56.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:56 smithi177 ceph-mon[31346]: pgmap v300: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:56.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:56 smithi105 ceph-mon[27916]: pgmap v300: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:58.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:46:58 smithi112 ceph-mon[31236]: pgmap v301: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:58.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:46:58 smithi177 ceph-mon[31346]: pgmap v301: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:46:58.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:46:58 smithi105 ceph-mon[27916]: pgmap v301: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:00.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:00 smithi112 ceph-mon[31236]: pgmap v302: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:00.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:00 smithi177 ceph-mon[31346]: pgmap v302: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:00.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:00 smithi105 ceph-mon[27916]: pgmap v302: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:02.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:02 smithi177 ceph-mon[31346]: pgmap v303: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:02.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:02 smithi105 ceph-mon[27916]: pgmap v303: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:02.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:02 smithi112 ceph-mon[31236]: pgmap v303: 97 pgs: 97 active+clean; 2.1 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:04.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:04 smithi177 ceph-mon[31346]: pgmap v304: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:04.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:04 smithi105 ceph-mon[27916]: pgmap v304: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:04.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:04 smithi112 ceph-mon[31236]: pgmap v304: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:06.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:06 smithi177 ceph-mon[31346]: pgmap v305: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:06.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:06 smithi105 ceph-mon[27916]: pgmap v305: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:06.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:06 smithi112 ceph-mon[31236]: pgmap v305: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:08.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:08 smithi177 ceph-mon[31346]: pgmap v306: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:08.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:08 smithi105 ceph-mon[27916]: pgmap v306: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:08.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:08 smithi112 ceph-mon[31236]: pgmap v306: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:10.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:10 smithi177 ceph-mon[31346]: pgmap v307: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:10.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:10 smithi105 ceph-mon[27916]: pgmap v307: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:10.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:10 smithi112 ceph-mon[31236]: pgmap v307: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:47:12.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:12 smithi177 ceph-mon[31346]: pgmap v308: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:12.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:12 smithi177 ceph-mon[31346]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:47:12.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:12 smithi105 ceph-mon[27916]: pgmap v308: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:12.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:12 smithi105 ceph-mon[27916]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:47:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:12 smithi112 ceph-mon[31236]: pgmap v308: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:47:12.709 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:12 smithi112 ceph-mon[31236]: osdmap e111: 12 total, 12 up, 12 in 2024-04-22T06:47:14.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:14 smithi177 ceph-mon[31346]: pgmap v310: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:14.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:14 smithi105 ceph-mon[27916]: pgmap v310: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:14.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:14 smithi112 ceph-mon[31236]: pgmap v310: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:16.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:16 smithi105 ceph-mon[27916]: pgmap v311: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:16.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:16 smithi105 ceph-mon[27916]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:47:16.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:16 smithi112 ceph-mon[31236]: pgmap v311: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:16.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:16 smithi112 ceph-mon[31236]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:47:16.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:16 smithi177 ceph-mon[31346]: pgmap v311: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:47:16.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:16 smithi177 ceph-mon[31346]: osdmap e112: 12 total, 12 up, 12 in 2024-04-22T06:47:18.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:18 smithi105 ceph-mon[27916]: pgmap v313: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:47:18.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:18 smithi112 ceph-mon[31236]: pgmap v313: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:47:18.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:18 smithi177 ceph-mon[31346]: pgmap v313: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:47:20.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:20 smithi105 ceph-mon[27916]: pgmap v314: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:47:20.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:20 smithi112 ceph-mon[31236]: pgmap v314: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:47:20.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:20 smithi177 ceph-mon[31346]: pgmap v314: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-22T06:47:22.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:22 smithi105 ceph-mon[27916]: pgmap v315: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-04-22T06:47:22.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:22 smithi112 ceph-mon[31236]: pgmap v315: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-04-22T06:47:22.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:22 smithi177 ceph-mon[31346]: pgmap v315: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2024-04-22T06:47:24.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:24 smithi105 ceph-mon[27916]: pgmap v316: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:24.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:24 smithi112 ceph-mon[31236]: pgmap v316: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:24.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:24 smithi177 ceph-mon[31346]: pgmap v316: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:26.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:26 smithi105 ceph-mon[27916]: pgmap v317: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:26.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:26 smithi112 ceph-mon[31236]: pgmap v317: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:26.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:26 smithi177 ceph-mon[31346]: pgmap v317: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:47:28.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:28 smithi105 ceph-mon[27916]: pgmap v318: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:47:28.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:28 smithi112 ceph-mon[31236]: pgmap v318: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:47:28.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:28 smithi177 ceph-mon[31346]: pgmap v318: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-04-22T06:47:30.542 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:30 smithi177 ceph-mon[31346]: pgmap v319: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:47:30.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:30 smithi105 ceph-mon[27916]: pgmap v319: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:47:30.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:30 smithi112 ceph-mon[31236]: pgmap v319: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-22T06:47:32.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:32 smithi105 ceph-mon[27916]: pgmap v320: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:32.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:32 smithi112 ceph-mon[31236]: pgmap v320: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:32.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:32 smithi177 ceph-mon[31346]: pgmap v320: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:34.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:34 smithi105 ceph-mon[27916]: pgmap v321: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:34.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:34 smithi112 ceph-mon[31236]: pgmap v321: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:34.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:34 smithi177 ceph-mon[31346]: pgmap v321: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:36.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:36 smithi105 ceph-mon[27916]: pgmap v322: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:36.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:36 smithi112 ceph-mon[31236]: pgmap v322: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:36.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:36 smithi177 ceph-mon[31346]: pgmap v322: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:38.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:38 smithi105 ceph-mon[27916]: pgmap v323: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:38.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:38 smithi112 ceph-mon[31236]: pgmap v323: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:38.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:38 smithi177 ceph-mon[31346]: pgmap v323: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:40.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:40 smithi105 ceph-mon[27916]: pgmap v324: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:40.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:40 smithi112 ceph-mon[31236]: pgmap v324: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:40.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:40 smithi177 ceph-mon[31346]: pgmap v324: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:42.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:42 smithi105 ceph-mon[27916]: pgmap v325: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:42.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:42 smithi112 ceph-mon[31236]: pgmap v325: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:42.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:42 smithi177 ceph-mon[31346]: pgmap v325: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:44.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:44 smithi105 ceph-mon[27916]: pgmap v326: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:44.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:44 smithi112 ceph-mon[31236]: pgmap v326: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:44.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:44 smithi177 ceph-mon[31346]: pgmap v326: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-22T06:47:45.968 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-22T06:47:46.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/3349013437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c01010f0 msgr2=0x7fe8c01014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.084 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/3349013437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c01010f0 0x7fe8c01014d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009900 tx=0x7fe8b402bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/3349013437 shutdown_connections 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/3349013437 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c0106170 0x7fe8c010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/3349013437 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0101aa0 0x7fe8c0105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/3349013437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c01010f0 0x7fe8c01014d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/3349013437 >> 172.21.15.105:0/3349013437 conn(0x7fe8c0077fd0 msgr2=0x7fe8c00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/3349013437 shutdown_connections 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.083+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/3349013437 wait complete. 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 Processor -- start 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 -- start start 2024-04-22T06:47:46.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c01010f0 0x7fe8c0197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 0x7fe8c0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c0111130 con 0x7fe8c019bf30 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c0110fb0 con 0x7fe8c0106170 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8c5f1e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c01112b0 con 0x7fe8c01010f0 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8bffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8bffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47428/0 (socket says 172.21.15.105:47428) 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8bffff640 1 -- 172.21.15.105:0/1167860431 learned_addr learned my addr 172.21.15.105:0/1167860431 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.084+0000 7fe8beffd640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 0x7fe8c0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 -- 172.21.15.105:0/1167860431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c01010f0 msgr2=0x7fe8c0197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c01010f0 0x7fe8c0197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 -- 172.21.15.105:0/1167860431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 msgr2=0x7fe8c0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 0x7fe8c0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 -- 172.21.15.105:0/1167860431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c019ca20 con 0x7fe8c019bf30 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8beffd640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 0x7fe8c0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bffff640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800b970 tx=0x7fe8b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8019070 con 0x7fe8c019bf30 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c019cd10 con 0x7fe8c019bf30 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8b8014440 con 0x7fe8c019bf30 2024-04-22T06:47:46.087 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80075d0 con 0x7fe8c019bf30 2024-04-22T06:47:46.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.085+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c01a4550 con 0x7fe8c019bf30 2024-04-22T06:47:46.088 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.086+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c004fce0 con 0x7fe8c019bf30 2024-04-22T06:47:46.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.087+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe8b801c070 con 0x7fe8c019bf30 2024-04-22T06:47:46.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.087+0000 7fe8bcff9640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 0x7fe8a80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.088+0000 7fe8bf7fe640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 0x7fe8a80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.088+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7fe8b80d0100 con 0x7fe8c019bf30 2024-04-22T06:47:46.089 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.088+0000 7fe8bf7fe640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 0x7fe8a80a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4005770 tx=0x7fe8b4000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.090 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.092+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8b809c4f0 con 0x7fe8c019bf30 2024-04-22T06:47:46.288 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.290+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe8c0109410 con 0x7fe8a80a3dc0 2024-04-22T06:47:46.296 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.298+0000 7fe8bcff9640 1 -- 172.21.15.105:0/1167860431 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fe8b4004110 con 0x7fe8a80a3dc0 2024-04-22T06:47:46.297 INFO:teuthology.orchestra.run.smithi105.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-22T00:00:00", "created": "2024-04-22T06:37:44", "first": "2024-04-22T06:38:00", "last": "2024-04-22T06:47:00", "last_pruned": "2024-04-22T06:47:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-22T06:47:46.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 msgr2=0x7fe8a80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 0x7fe8a80a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4005770 tx=0x7fe8b4000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 msgr2=0x7fe8c019c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800b970 tx=0x7fe8b800be30 comp rx=0 tx=0).stop 2024-04-22T06:47:46.299 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 shutdown_connections 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fe8a80a3dc0 0x7fe8a80a6280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe8c019bf30 0x7fe8c019c3e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c0106170 0x7fe8c0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 --2- 172.21.15.105:0/1167860431 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8c01010f0 0x7fe8c0197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.300+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 >> 172.21.15.105:0/1167860431 conn(0x7fe8c0077fd0 msgr2=0x7fe8c00ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.301+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 shutdown_connections 2024-04-22T06:47:46.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.301+0000 7fe8c5f1e640 1 -- 172.21.15.105:0/1167860431 wait complete. 2024-04-22T06:47:46.309 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-22T06:47:46.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 -- 172.21.15.105:0/321948689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 msgr2=0x7fabf0111000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 --2- 172.21.15.105:0/321948689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf0111000 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fabe8009ca0 tx=0x7fabe802c120 comp rx=0 tx=0).stop 2024-04-22T06:47:46.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 -- 172.21.15.105:0/321948689 shutdown_connections 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 --2- 172.21.15.105:0/321948689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf0111000 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 --2- 172.21.15.105:0/321948689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf01009d0 0x7fabf0100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 --2- 172.21.15.105:0/321948689 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf0103090 0x7fabf0100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 -- 172.21.15.105:0/321948689 >> 172.21.15.105:0/321948689 conn(0x7fabf0077390 msgr2=0x7fabf00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 -- 172.21.15.105:0/321948689 shutdown_connections 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.411+0000 7fabf617e640 1 -- 172.21.15.105:0/321948689 wait complete. 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 Processor -- start 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 -- start start 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 0x7fabf01a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.413 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf0103090 0x7fabf01a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0114680 con 0x7fabf010ec10 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0114500 con 0x7fabf0103090 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabf617e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0114800 con 0x7fabf01009d0 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47442/0 (socket says 172.21.15.105:47442) 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 -- 172.21.15.105:0/223525012 learned_addr learned my addr 172.21.15.105:0/223525012 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeeffd640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf0103090 0x7fabf01a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.414 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabef7fe640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 0x7fabf01a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 -- 172.21.15.105:0/223525012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 msgr2=0x7fabf01a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 0x7fabf01a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 -- 172.21.15.105:0/223525012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf0103090 msgr2=0x7fabf01a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf0103090 0x7fabf01a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabeffff640 1 -- 172.21.15.105:0/223525012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf019b010 con 0x7fabf010ec10 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.412+0000 7fabef7fe640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 0x7fabf01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabeffff640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fabe802c730 tx=0x7fabe8035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8039040 con 0x7fabf010ec10 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabe8044070 con 0x7fabf010ec10 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf019b2a0 con 0x7fabf010ec10 2024-04-22T06:47:46.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8004570 con 0x7fabf010ec10 2024-04-22T06:47:46.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.413+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf01a7520 con 0x7fabf010ec10 2024-04-22T06:47:46.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.414+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf0101740 con 0x7fabf010ec10 2024-04-22T06:47:46.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.415+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fabe8004710 con 0x7fabf010ec10 2024-04-22T06:47:46.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.416+0000 7fabecff9640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 0x7fabd80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.416+0000 7fabef7fe640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 0x7fabd80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.416+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7fabe80f3430 con 0x7fabf010ec10 2024-04-22T06:47:46.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.416+0000 7fabef7fe640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 0x7fabd80a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fabe400a8b0 tx=0x7fabe4008040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.420+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabe80bf820 con 0x7fabf010ec10 2024-04-22T06:47:46.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.620+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 --> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fabf006a580 con 0x7fabd80a3dc0 2024-04-22T06:47:46.627 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.629+0000 7fabecff9640 1 -- 172.21.15.105:0/223525012 <== mgr.15357 v2:172.21.15.105:6800/971331810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fabf006a580 con 0x7fabd80a3dc0 2024-04-22T06:47:46.628 INFO:teuthology.orchestra.run.smithi105.stdout:/ 1m 6m3h 2024-04-22T06:47:46.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 msgr2=0x7fabd80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 0x7fabd80a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fabe400a8b0 tx=0x7fabe4008040 comp rx=0 tx=0).stop 2024-04-22T06:47:46.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 msgr2=0x7fabf019a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fabe802c730 tx=0x7fabe8035dd0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 shutdown_connections 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fabd80a3dc0 0x7fabd80a6280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fabf010ec10 0x7fabf019a840 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabf0103090 0x7fabf01a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 --2- 172.21.15.105:0/223525012 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fabf01009d0 0x7fabf01a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 >> 172.21.15.105:0/223525012 conn(0x7fabf0077390 msgr2=0x7fabf010ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 shutdown_connections 2024-04-22T06:47:46.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.631+0000 7fabf617e640 1 -- 172.21.15.105:0/223525012 wait complete. 2024-04-22T06:47:46.639 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:46 smithi105 ceph-mon[27916]: pgmap v327: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:46.640 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-22T06:47:46.658 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-22T06:47:46.659 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_86f7587a5a09af35f0895e1a2d08527638fae697/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Command failed on smithi105 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls' 2024-04-22T06:47:46.659 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-22T06:47:46.670 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:47:46.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:46 smithi112 ceph-mon[31236]: pgmap v327: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:46.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:46 smithi177 ceph-mon[31346]: pgmap v327: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:46.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 -- 172.21.15.105:0/2531153308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f41031d0 msgr2=0x7fa5f41035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.771 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 --2- 172.21.15.105:0/2531153308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f41031d0 0x7fa5f41035b0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009900 tx=0x7fa5e802bce0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 -- 172.21.15.105:0/2531153308 shutdown_connections 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 --2- 172.21.15.105:0/2531153308 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5f410c5a0 0x7fa5f410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 --2- 172.21.15.105:0/2531153308 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f4103af0 0x7fa5f410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 --2- 172.21.15.105:0/2531153308 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f41031d0 0x7fa5f41035b0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 -- 172.21.15.105:0/2531153308 >> 172.21.15.105:0/2531153308 conn(0x7fa5f40771d0 msgr2=0x7fa5f40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 -- 172.21.15.105:0/2531153308 shutdown_connections 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.770+0000 7fa5fae11640 1 -- 172.21.15.105:0/2531153308 wait complete. 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 Processor -- start 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 -- start start 2024-04-22T06:47:46.772 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f41031d0 0x7fa5f419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5f4103af0 0x7fa5f419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f4112240 con 0x7fa5f410c5a0 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f41120c0 con 0x7fa5f41031d0 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f41123c0 con 0x7fa5f4103af0 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47468/0 (socket says 172.21.15.105:47468) 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 -- 172.21.15.105:0/3327629444 learned_addr learned my addr 172.21.15.105:0/3327629444 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f8bad640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f41031d0 0x7fa5f419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 -- 172.21.15.105:0/3327629444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5f4103af0 msgr2=0x7fa5f419e950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:46.773 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5f4103af0 0x7fa5f419e950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 -- 172.21.15.105:0/3327629444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f41031d0 msgr2=0x7fa5f419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f41031d0 0x7fa5f419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 -- 172.21.15.105:0/3327629444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f4198d40 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f93ae640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b9e0 tx=0x7fa5ec00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec019070 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5ec014440 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f4199030 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.771+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0125a0 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.772+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f41a52d0 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.773+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f4069230 con 0x7fa5f410c5a0 2024-04-22T06:47:46.774 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.774+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fa5ec01c070 con 0x7fa5f410c5a0 2024-04-22T06:47:46.775 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.774+0000 7fa5f1ffb640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 0x7fa5e40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:46.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.774+0000 7fa5f8bad640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 0x7fa5e40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:46.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.774+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7fa5ec0cff50 con 0x7fa5f410c5a0 2024-04-22T06:47:46.776 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.775+0000 7fa5f8bad640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 0x7fa5e40a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8006010 tx=0x7fa5e8005e60 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:46.777 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.779+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5ec09bcf0 con 0x7fa5f410c5a0 2024-04-22T06:47:46.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.957+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5f4107100 con 0x7fa5f410c5a0 2024-04-22T06:47:46.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.959+0000 7fa5f1ffb640 1 -- 172.21.15.105:0/3327629444 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) ==== 74+0+24654 (secure 0 0 0) 0x7fa5ec09fff0 con 0x7fa5f410c5a0 2024-04-22T06:47:46.958 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:47:46.958 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":112,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:47:16.145790+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":111,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6837/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6839/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.177:6838/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.112:6836/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.177:6837/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000","172.21.15.177:6832/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6833/583171810":"2024-04-23T06:39:27.429757+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.112:6836/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.112:6837/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.112:6837/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.105:6840/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6841/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6841/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6838/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6838/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.177:6839/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.105:6840/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6839/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6836/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6833/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6834/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.112:6835/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.177:6836/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.112:6832/583171810":"2024-04-23T06:39:27.429757+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 msgr2=0x7fa5e40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 0x7fa5e40a6260 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8006010 tx=0x7fa5e8005e60 comp rx=0 tx=0).stop 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 msgr2=0x7fa5f41985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b9e0 tx=0x7fa5ec00bea0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 shutdown_connections 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa5e40a3da0 0x7fa5e40a6260 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5f410c5a0 0x7fa5f41985c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.960 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa5f4103af0 0x7fa5f419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 --2- 172.21.15.105:0/3327629444 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa5f41031d0 0x7fa5f419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:46.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 >> 172.21.15.105:0/3327629444 conn(0x7fa5f40771d0 msgr2=0x7fa5f40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:46.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 shutdown_connections 2024-04-22T06:47:46.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:46.961+0000 7fa5fae11640 1 -- 172.21.15.105:0/3327629444 wait complete. 2024-04-22T06:47:46.971 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 111, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:47:46.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- 172.21.15.105:0/2946867184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 msgr2=0x7f906010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 --2- 172.21.15.105:0/2946867184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f906010d440 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f905800aa10 tx=0x7f905802cf70 comp rx=0 tx=0).stop 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- 172.21.15.105:0/2946867184 shutdown_connections 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 --2- 172.21.15.105:0/2946867184 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f906010d440 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 --2- 172.21.15.105:0/2946867184 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 0x7f9060100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.076 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 --2- 172.21.15.105:0/2946867184 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 0x7f9060106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- 172.21.15.105:0/2946867184 >> 172.21.15.105:0/2946867184 conn(0x7f9060077410 msgr2=0x7f90600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- 172.21.15.105:0/2946867184 shutdown_connections 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- 172.21.15.105:0/2946867184 wait complete. 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 Processor -- start 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.075+0000 7f9065d9d640 1 -- start start 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 0x7f9060198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 0x7f906019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f906010fff0 con 0x7f9060100f80 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f906010fe70 con 0x7f90601005c0 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9060110170 con 0x7f9060106680 2024-04-22T06:47:47.077 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47490/0 (socket says 172.21.15.105:47490) 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 -- 172.21.15.105:0/3743177840 learned_addr learned my addr 172.21.15.105:0/3743177840 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905f7fe640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 0x7f9060198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905ffff640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 0x7f906019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 -- 172.21.15.105:0/3743177840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 msgr2=0x7f906019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 0x7f906019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 -- 172.21.15.105:0/3743177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 msgr2=0x7f9060198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 0x7f9060198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 -- 172.21.15.105:0/3743177840 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906019d670 con 0x7f9060100f80 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905effd640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f904800cce0 tx=0x7f904800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048004b20 con 0x7f9060100f80 2024-04-22T06:47:47.078 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906019d960 con 0x7f9060100f80 2024-04-22T06:47:47.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9048004cc0 con 0x7f9060100f80 2024-04-22T06:47:47.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904800f6b0 con 0x7f9060100f80 2024-04-22T06:47:47.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.076+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90601a51f0 con 0x7f9060100f80 2024-04-22T06:47:47.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.077+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90600691c0 con 0x7f9060100f80 2024-04-22T06:47:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.078+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f9048005000 con 0x7f9060100f80 2024-04-22T06:47:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.079+0000 7f905cff9640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.079+0000 7f905f7fe640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.079+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7f90480d0c70 con 0x7f9060100f80 2024-04-22T06:47:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.079+0000 7f905f7fe640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 0x7f904c0a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f905000ab60 tx=0x7f905000a2d0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.081 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.083+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9048014030 con 0x7f9060100f80 2024-04-22T06:47:47.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.260+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f906010aa60 con 0x7f9060100f80 2024-04-22T06:47:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.261+0000 7f905cff9640 1 -- 172.21.15.105:0/3743177840 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) ==== 93+0+10 (secure 0 0 0) 0x7f904809cf50 con 0x7f9060100f80 2024-04-22T06:47:47.259 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:47:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 msgr2=0x7f904c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 0x7f904c0a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f905000ab60 tx=0x7f905000a2d0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 msgr2=0x7f9060198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f904800cce0 tx=0x7f904800a620 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 shutdown_connections 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f904c0a3d20 0x7f904c0a61e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9060106680 0x7f906019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9060100f80 0x7f9060198bc0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 --2- 172.21.15.105:0/3743177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f90601005c0 0x7f9060198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.262 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 >> 172.21.15.105:0/3743177840 conn(0x7f9060077410 msgr2=0x7f9060104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 shutdown_connections 2024-04-22T06:47:47.263 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.263+0000 7f9065d9d640 1 -- 172.21.15.105:0/3743177840 wait complete. 2024-04-22T06:47:47.272 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:47:47.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 -- 172.21.15.105:0/1902001822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b010c510 msgr2=0x7fa7b010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/1902001822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b010c510 0x7fa7b010e9b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8009ca0 tx=0x7fa7a802c230 comp rx=0 tx=0).stop 2024-04-22T06:47:47.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 -- 172.21.15.105:0/1902001822 shutdown_connections 2024-04-22T06:47:47.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/1902001822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b010c510 0x7fa7b010e9b0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.377 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/1902001822 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7b0103af0 0x7fa7b010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/1902001822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b01031d0 0x7fa7b01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 -- 172.21.15.105:0/1902001822 >> 172.21.15.105:0/1902001822 conn(0x7fa7b0077f50 msgr2=0x7fa7b0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 -- 172.21.15.105:0/1902001822 shutdown_connections 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.376+0000 7fa7b68c1640 1 -- 172.21.15.105:0/1902001822 wait complete. 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 Processor -- start 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 -- start start 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b0103af0 0x7fa7b019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7b010c510 0x7fa7b01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.378 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b0112220 con 0x7fa7b01031d0 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01120a0 con 0x7fa7b0103af0 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7b68c1640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01123a0 con 0x7fa7b010c510 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47520/0 (socket says 172.21.15.105:47520) 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.377+0000 7fa7affff640 1 -- 172.21.15.105:0/2659901128 learned_addr learned my addr 172.21.15.105:0/2659901128 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7af7fe640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b0103af0 0x7fa7b019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 -- 172.21.15.105:0/2659901128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7b010c510 msgr2=0x7fa7b01985a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7b010c510 0x7fa7b01985a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 -- 172.21.15.105:0/2659901128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b0103af0 msgr2=0x7fa7b019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b0103af0 0x7fa7b019e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 -- 172.21.15.105:0/2659901128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b0198d20 con 0x7fa7b01031d0 2024-04-22T06:47:47.379 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7affff640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d950 tx=0x7fa7a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0017070 con 0x7fa7b01031d0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b0199010 con 0x7fa7b01031d0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7a0009c30 con 0x7fa7b01031d0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a000f620 con 0x7fa7b01031d0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.378+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b01a52b0 con 0x7fa7b01031d0 2024-04-22T06:47:47.380 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.379+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b0068d70 con 0x7fa7b01031d0 2024-04-22T06:47:47.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.380+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fa7a0005030 con 0x7fa7b01031d0 2024-04-22T06:47:47.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.381+0000 7fa7ad7fa640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 0x7fa7980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.381+0000 7fa7af7fe640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 0x7fa7980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.381+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7fa7a00d0890 con 0x7fa7b01031d0 2024-04-22T06:47:47.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.381+0000 7fa7af7fe640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 0x7fa7980a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7b010e340 tx=0x7fa7a4008040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.385+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7a009cb70 con 0x7fa7b01031d0 2024-04-22T06:47:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.559+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7b0199ad0 con 0x7fa7b01031d0 2024-04-22T06:47:47.558 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.560+0000 7fa7ad7fa640 1 -- 172.21.15.105:0/2659901128 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) ==== 104+0+11 (secure 0 0 0) 0x7fa7b0199ad0 con 0x7fa7b01031d0 2024-04-22T06:47:47.558 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 msgr2=0x7fa7980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 0x7fa7980a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7b010e340 tx=0x7fa7a4008040 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 msgr2=0x7fa7b019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000d950 tx=0x7fa7a000de10 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 shutdown_connections 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7980a3d20 0x7fa7980a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa7b010c510 0x7fa7b01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7b0103af0 0x7fa7b019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 --2- 172.21.15.105:0/2659901128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7b01031d0 0x7fa7b019e3f0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 >> 172.21.15.105:0/2659901128 conn(0x7fa7b0077f50 msgr2=0x7fa7b00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 shutdown_connections 2024-04-22T06:47:47.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.562+0000 7fa7b68c1640 1 -- 172.21.15.105:0/2659901128 wait complete. 2024-04-22T06:47:47.570 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:47 smithi105 ceph-mon[27916]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.570 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:47 smithi105 ceph-mon[27916]: from='client.16584 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.570 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3327629444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:47:47.570 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:47 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/3743177840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:47:47.571 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 -- 172.21.15.105:0/1120993438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06b210 msgr2=0x7f7c1c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/1120993438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06b210 0x7f7c1c106b90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009900 tx=0x7f7c0c02bc30 comp rx=0 tx=0).stop 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 -- 172.21.15.105:0/1120993438 shutdown_connections 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/1120993438 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c107270 0x7f7c1c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/1120993438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06b210 0x7f7c1c106b90 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/1120993438 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c06a860 0x7f7c1c06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 -- 172.21.15.105:0/1120993438 >> 172.21.15.105:0/1120993438 conn(0x7f7c1c077eb0 msgr2=0x7f7c1c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 -- 172.21.15.105:0/1120993438 shutdown_connections 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 -- 172.21.15.105:0/1120993438 wait complete. 2024-04-22T06:47:47.675 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.674+0000 7f7c21dbd640 1 Processor -- start 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 -- start start 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 0x7f7c1c19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 0x7f7c1c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c112210 con 0x7f7c1c06a860 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c112090 con 0x7f7c1c198610 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c21dbd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c112390 con 0x7f7c1c107270 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47532/0 (socket says 172.21.15.105:47532) 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 -- 172.21.15.105:0/783565400 learned_addr learned my addr 172.21.15.105:0/783565400 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1bfff640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 0x7f7c1c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1affd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 0x7f7c1c19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 -- 172.21.15.105:0/783565400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 msgr2=0x7f7c1c19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.676 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 0x7f7c1c19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 -- 172.21.15.105:0/783565400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 msgr2=0x7f7c1c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 0x7f7c1c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 -- 172.21.15.105:0/783565400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c199130 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1affd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 0x7f7c1c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1bfff640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 0x7f7c1c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c1b7fe640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000b4a0 tx=0x7f7c1000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10014070 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.675+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c10010ac0 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.676+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c199420 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.676+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10017c30 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.676+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1a5670 con 0x7f7c1c06a860 2024-04-22T06:47:47.677 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.677+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c0691c0 con 0x7f7c1c06a860 2024-04-22T06:47:47.678 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.678+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f7c10017070 con 0x7f7c1c06a860 2024-04-22T06:47:47.678 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.678+0000 7f7c18ff9640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 0x7f7c040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.678 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.678+0000 7f7c1affd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 0x7f7c040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.678 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.678+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7f7c100d0250 con 0x7f7c1c06a860 2024-04-22T06:47:47.679 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.679+0000 7f7c1affd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 0x7f7c040a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006010 tx=0x7f7c0c000bc0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.681 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.683+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c1009c640 con 0x7f7c1c06a860 2024-04-22T06:47:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:47 smithi112 ceph-mon[31236]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:47 smithi112 ceph-mon[31236]: from='client.16584 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3327629444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:47:47.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:47 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/3743177840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:47:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:47 smithi177 ceph-mon[31346]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:47 smithi177 ceph-mon[31346]: from='client.16584 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T06:47:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3327629444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:47:47.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:47 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/3743177840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:47:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.863+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7c1c06b210 con 0x7f7c1c06a860 2024-04-22T06:47:47.862 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.864+0000 7f7c18ff9640 1 -- 172.21.15.105:0/783565400 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) ==== 100+0+11 (secure 0 0 0) 0x7f7c1009bf40 con 0x7f7c1c06a860 2024-04-22T06:47:47.862 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 msgr2=0x7f7c040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 0x7f7c040a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006010 tx=0x7f7c0c000bc0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 msgr2=0x7f7c1c19e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000b4a0 tx=0x7f7c1000b960 comp rx=0 tx=0).stop 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 shutdown_connections 2024-04-22T06:47:47.864 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f7c040a3d20 0x7f7c040a61e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1c198610 0x7f7c1c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7c1c107270 0x7f7c1c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 --2- 172.21.15.105:0/783565400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c1c06a860 0x7f7c1c19e460 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 >> 172.21.15.105:0/783565400 conn(0x7f7c1c077eb0 msgr2=0x7f7c1c108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 shutdown_connections 2024-04-22T06:47:47.865 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.866+0000 7f7c21dbd640 1 -- 172.21.15.105:0/783565400 wait complete. 2024-04-22T06:47:47.875 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.974+0000 7f2823dca640 1 -- 172.21.15.105:0/2317013855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c10c520 msgr2=0x7f281c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.974+0000 7f2823dca640 1 --2- 172.21.15.105:0/2317013855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c10c520 0x7f281c10e9c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2814009900 tx=0x7f281402bd60 comp rx=0 tx=0).stop 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.974+0000 7f2823dca640 1 -- 172.21.15.105:0/2317013855 shutdown_connections 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- 172.21.15.105:0/2317013855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c10c520 0x7f281c10e9c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- 172.21.15.105:0/2317013855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c103b00 0x7f281c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- 172.21.15.105:0/2317013855 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 0x7f281c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- 172.21.15.105:0/2317013855 >> 172.21.15.105:0/2317013855 conn(0x7f281c077f50 msgr2=0x7f281c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- 172.21.15.105:0/2317013855 shutdown_connections 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- 172.21.15.105:0/2317013855 wait complete. 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 Processor -- start 2024-04-22T06:47:47.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- start start 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 0x7f281c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c10c520 0x7f281c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1121b0 con 0x7f281c103b00 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f281c112030 con 0x7f281c10c520 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2823dca640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f281c112330 con 0x7f281c1031e0 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.975+0000 7f2821365640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47552/0 (socket says 172.21.15.105:47552) 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 -- 172.21.15.105:0/1941323215 learned_addr learned my addr 172.21.15.105:0/1941323215 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821b66640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 0x7f281c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2822367640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c10c520 0x7f281c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 -- 172.21.15.105:0/1941323215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 msgr2=0x7f281c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 0x7f281c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 -- 172.21.15.105:0/1941323215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c10c520 msgr2=0x7f281c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:47.977 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c10c520 0x7f281c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 -- 172.21.15.105:0/1941323215 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c19d6f0 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2821365640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f280c00d950 tx=0x7f280c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c017070 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c19d9e0 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280c009c30 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c00f620 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.976+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c1a5270 con 0x7f281c103b00 2024-04-22T06:47:47.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.977+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f281c068d70 con 0x7f281c103b00 2024-04-22T06:47:47.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.978+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f280c010460 con 0x7f281c103b00 2024-04-22T06:47:47.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.979+0000 7f2812ffd640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 0x7f28080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:47:47.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.979+0000 7f2821b66640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 0x7f28080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:47:47.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.979+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9487+0+0 (secure 0 0 0) 0x7f280c0d06c0 con 0x7f281c103b00 2024-04-22T06:47:47.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.979+0000 7f2821b66640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 0x7f28080a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2818005fd0 tx=0x7f2818009680 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:47:47.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:47.984+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f280c009de0 con 0x7f281c103b00 2024-04-22T06:47:48.211 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.213+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f281c107700 con 0x7f281c103b00 2024-04-22T06:47:48.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.221+0000 7f2812ffd640 1 -- 172.21.15.105:0/1941323215 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746069 (secure 0 0 0) 0x7f280c09c350 con 0x7f281c103b00 2024-04-22T06:47:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:47:48.230 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 26, 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "active_gid": 15357, 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "active_name": "x", 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "active_addrs": { 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6800", 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 971331810 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v1", 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6801", 2024-04-22T06:47:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 971331810 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "active_addr": "172.21.15.105:6801/971331810", 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "active_change": "2024-04-22T06:37:36.925202+0000", 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "active_mgr_features": 4540701547738038271, 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "available": true, 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "standbys": [ 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "gid": 24905, 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "y", 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "available_modules": [ 2024-04-22T06:47:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "alerts", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:47:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_destination": { 2024-04-22T06:47:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_destination", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_from_name": { 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_from_name", 2024-04-22T06:47:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "Ceph", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email From: name", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_host": { 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_host", 2024-04-22T06:47:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP server", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_password": { 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_password", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Password to authenticate with", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_port": { 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_port", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "465", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP port", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_sender": { 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_sender", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_ssl": { 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_ssl", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_user": { 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_user", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "User to authenticate as", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "balancer", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "active": { 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "active", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:47:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_time": { 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_time", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0000", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_weekday": { 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_weekday", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_max_iterations": { 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "25", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "250", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_metrics": { 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_metrics", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_step": { 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_step", 2024-04-22T06:47:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.5", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0.001", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "0.999", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "end_time": { 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_time", 2024-04-22T06:47:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2359", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "end_weekday": { 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_weekday", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "min_score": { 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_score", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "upmap", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-compat", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "none", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "read", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap", 2024-04-22T06:47:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap-read" 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Balancer mode", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_ids": { 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_ids", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_deviation": { 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_deviation", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_optimizations": { 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm", 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_down_multiplier": { 2024-04-22T06:47:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_down_multiplier", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_refresh_rate": { 2024-04-22T06:47:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_refresh_rate", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "20", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_starting_port": { 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_starting_port", 2024-04-22T06:47:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4721", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_ptrace": { 2024-04-22T06:47:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_ptrace", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_interval": { 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_interval", 2024-04-22T06:47:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:47:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.7", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm_log_destination": { 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm_log_destination", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "file", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "file,syslog", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "syslog" 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "cgroups_split": { 2024-04-22T06:47:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cgroups_split", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "config_checks_enabled": { 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_checks_enabled", 2024-04-22T06:47:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "config_dashboard": { 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_dashboard", 2024-04-22T06:47:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_alertmanager": { 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_alertmanager", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_base": { 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_base", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:47:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_elasticsearch": { 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:47:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "elasticsearch container image", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_grafana": { 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_grafana", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:47:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_haproxy": { 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_haproxy", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "HAproxy container image", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_agent": { 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_collector": { 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:47:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_query": { 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger query container image", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_keepalived": { 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_keepalived", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Keepalived container image", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_loki": { 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_loki", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Loki container image", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_node_exporter": { 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_node_exporter", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_nvmeof": { 2024-04-22T06:47:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_nvmeof", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Nvme-of container image", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_prometheus": { 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_prometheus", 2024-04-22T06:47:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_promtail": { 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_promtail", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Promtail container image", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_samba": { 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_samba", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_snmp_gateway": { 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:47:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "container_init": { 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_init", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "daemon_cache_timeout": { 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "900", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:47:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "default_registry": { 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_registry", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "device_cache_timeout": { 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_cache_timeout", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1800", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "device_enhanced_scan": { 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_enhanced_scan", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "facts_cache_timeout": { 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "facts_cache_timeout", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "grafana_dashboards_path": { 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:47:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "host_check_interval": { 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "host_check_interval", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "hw_monitoring": { 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hw_monitoring", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "inventory_list_all": { 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "inventory_list_all", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "log_refresh_metadata": { 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_refresh_metadata", 2024-04-22T06:47:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "*", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "max_count_per_host": { 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_count_per_host", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd_draining_count": { 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_osd_draining_count", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "migration_current": { 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "migration_current", 2024-04-22T06:47:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "internal - do not modify", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "root", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm-package", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "root" 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "oob_default_addr": { 2024-04-22T06:47:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "oob_default_addr", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "169.254.1.1", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "prometheus_alerts_path": { 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_insecure": { 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_insecure", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_password": { 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_password", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_url": { 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_url", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_username": { 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_username", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "secure_monitoring_stack": { 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "service_discovery_port": { 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "service_discovery_port", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8765", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "ssh_config_file": { 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssh_config_file", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:47:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "use_agent": { 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_agent", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "use_repo_digest": { 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_repo_digest", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_failed_host_check": { 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_daemons": { 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_hosts": { 2024-04-22T06:47:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crash", 2024-04-22T06:47:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "retain_interval": { 2024-04-22T06:47:48.329 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retain_interval", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "31536000", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_recent_interval": { 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_recent_interval", 2024-04-22T06:47:48.330 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1209600", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dashboard", 2024-04-22T06:47:48.331 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.333 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:47:48.334 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:47:48.335 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.336 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.337 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.338 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.339 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.340 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.341 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.342 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.343 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.344 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.345 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_URL": { 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:47:48.348 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:47:48.349 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.350 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.351 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "[]", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.352 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "{}", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.353 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.354 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.355 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.357 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:47:48.358 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.359 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.360 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.361 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.362 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.363 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.364 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.365 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "45", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.367 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:47:48.368 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:47:48.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:47:48.370 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.372 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.373 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "cross_origin_url": { 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cross_origin_url", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.374 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "crt_file": { 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crt_file", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.375 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "debug": { 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "debug", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.376 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "jwt_token_ttl": { 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "jwt_token_ttl", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "28800", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.377 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.378 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.381 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.381 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.381 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.381 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.381 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.382 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.383 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.384 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.385 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.386 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "motd": { 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "motd", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "The message of the day", 2024-04-22T06:47:48.387 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8080", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl_server_port": { 2024-04-22T06:47:48.391 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl_server_port", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8443", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.392 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "redirect", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect" 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:48.393 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "url_prefix": { 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url_prefix", 2024-04-22T06:47:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "devicehealth", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_monitoring": { 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_monitoring", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "monitor device health metrics", 2024-04-22T06:47:48.396 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.397 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.398 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.399 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.400 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.401 INFO:teuthology.orchestra.run.smithi105.stdout: "mark_out_threshold": { 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mark_out_threshold", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2419200", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.402 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": { 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_name", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "device_health_metrics", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:47:48.403 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "retention_period": { 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retention_period", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15552000", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.404 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_frequency": { 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_frequency", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.405 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "self_heal": { 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "self_heal", 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.406 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.407 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_threshold": { 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_threshold", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.408 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7257600", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "diskprediction_local", 2024-04-22T06:47:48.409 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.410 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.411 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.412 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.413 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "predict_interval": { 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predict_interval", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.414 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "predictor_model": { 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predictor_model", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.415 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "prophetstor", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.416 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "influx", 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": false, 2024-04-22T06:47:48.417 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "batch_size": { 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "batch_size", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5000", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "database": { 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "database", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "ceph", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "hostname": { 2024-04-22T06:47:48.419 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hostname", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.420 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "30", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "5", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.421 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.422 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.424 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.425 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "password": { 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "password", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.426 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "port": { 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "port", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8086", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.427 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server port", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "false", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.428 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "threads": { 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threads", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "32", 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.429 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "username": { 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "username", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.430 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "verify_ssl": { 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "verify_ssl", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "true", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:48.431 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "insights", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.432 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.434 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.435 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.436 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "iostat", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.437 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.438 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.440 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.441 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "k8sevents", 2024-04-22T06:47:48.442 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_event_retention_days": { 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.443 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "config_check_secs": { 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_check_secs", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "10", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.444 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.445 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.448 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.449 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "localpool", 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "failure_domain": { 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "failure_domain", 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.450 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "host", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.451 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.452 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.453 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.454 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.455 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": { 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_size", 2024-04-22T06:47:48.456 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "num_rep": { 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "num_rep", 2024-04-22T06:47:48.457 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": { 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_num", 2024-04-22T06:47:48.458 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "128", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "prefix": { 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prefix", 2024-04-22T06:47:48.459 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "subtree": { 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "subtree", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "rack", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mds_autoscaler", 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.461 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.462 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.463 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.464 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.465 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirroring", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.467 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.468 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.469 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.470 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.471 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "nfs", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.472 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.473 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.474 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.475 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.476 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.477 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "fail_fs": { 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "fail_fs", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.479 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.480 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.481 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.482 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator": { 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:48.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "rook", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "test_orchestrator" 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Orchestrator backend", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.484 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_perf_query", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.485 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.486 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.487 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.488 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_support", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.490 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.491 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.492 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.493 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.494 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_autoscaler", 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.495 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.496 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.497 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.499 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.500 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "threshold": { 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threshold", 2024-04-22T06:47:48.501 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1.0", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "scaling threshold", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "progress", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_pg_recovery_event": { 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.503 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.504 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.505 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.506 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.507 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.508 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "max_completed_events": { 2024-04-22T06:47:48.509 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_completed_events", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "50", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.510 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.511 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus", 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "cache": { 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cache", 2024-04-22T06:47:48.512 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "exclude_perf_counters": { 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "exclude_perf_counters", 2024-04-22T06:47:48.513 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.514 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.515 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.516 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.517 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.518 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools": { 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.519 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "300", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.520 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_interval": { 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_interval", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15.0", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.521 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:47:48.522 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "9283", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.523 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "stale_cache_strategy": { 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stale_cache_strategy", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "log", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "default", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "default", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "error" 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.525 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.526 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_support", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.527 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.529 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.530 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.531 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_snap_create": { 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.532 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.533 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "trash_purge_schedule": { 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "trash_purge_schedule", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.534 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "restful", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_auth": { 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_auth", 2024-04-22T06:47:48.535 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.536 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.537 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.538 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.540 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.541 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.542 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.543 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rgw", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.544 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.548 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rook", 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.549 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.551 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.552 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.553 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "storage_class": { 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "storage_class", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.554 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "local", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "selftest", 2024-04-22T06:47:48.555 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.556 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.557 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.558 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "roption1": { 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption1", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.560 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "roption2": { 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption2", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.561 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "xyz", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption1": { 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption1", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.562 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption2": { 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption2", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.563 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption3": { 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption3", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.564 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption4": { 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption4", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.565 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption5": { 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption5", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.566 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption6": { 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption6", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.567 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption7": { 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption7", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "42", 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.568 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "testkey": { 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testkey", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.569 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "testlkey": { 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testlkey", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.570 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "testnewline": { 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testnewline", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.571 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snap_schedule", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_m_granularity": { 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_m_granularity", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.572 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "dump_on_update": { 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dump_on_update", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.573 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.574 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.575 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.576 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.577 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stats", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.579 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.580 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.581 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.582 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.583 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "status", 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.584 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.585 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.586 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.587 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.588 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telegraf", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "address": { 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "address", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.590 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.591 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.592 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.593 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.594 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telemetry", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_basic": { 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_basic", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.597 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_crash": { 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_crash", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.598 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_device": { 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_device", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:47:48.599 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_ident": { 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_ident", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.600 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_perf": { 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_perf", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.601 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "contact": { 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "contact", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.602 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": { 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "description", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.603 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "device_url": { 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_url", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.604 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.605 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "24", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "8", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.606 INFO:teuthology.orchestra.run.smithi105.stdout: "last_opt_revision": { 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "last_opt_revision", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard": { 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard", 2024-04-22T06:47:48.607 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard_description": { 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard_description", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.608 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.609 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.610 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.611 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.613 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "organization": { 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "organization", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.614 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "proxy": { 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "proxy", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.615 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "url": { 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.616 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "test_orchestrator", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.617 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.618 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.619 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.620 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.621 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "volumes", 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.622 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.623 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.624 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.625 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.626 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.627 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_clones": { 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_clones", 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.628 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "periodic_async_work": { 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "periodic_async_work", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Periodically check for async work", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_delay": { 2024-04-22T06:47:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:47:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "discovery_interval": { 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "discovery_interval", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "100", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.633 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "identifier": { 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "identifier", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.634 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.635 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.636 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.637 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.638 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.639 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.640 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_host": { 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_host", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.641 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_port": { 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_port", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.642 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10051", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_sender": { 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_sender", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.643 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "gid": 24913, 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "z", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_features": 4540701547738038271, 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "available_modules": [ 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "alerts", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.645 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.646 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.647 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.648 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.649 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.650 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.651 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_destination": { 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_destination", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.652 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_from_name": { 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_from_name", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "Ceph", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email From: name", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.653 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_host": { 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_host", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP server", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.654 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_password": { 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_password", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Password to authenticate with", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.655 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_port": { 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_port", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "465", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP port", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.656 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_sender": { 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_sender", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.657 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_ssl": { 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_ssl", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.658 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_user": { 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_user", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "User to authenticate as", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.659 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "balancer", 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "active": { 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "active", 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.660 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_time": { 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_time", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.661 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0000", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_weekday": { 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_weekday", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.662 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_max_iterations": { 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.663 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "25", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "250", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_metrics": { 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_metrics", 2024-04-22T06:47:48.664 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_step": { 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_step", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.665 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.5", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0.001", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "0.999", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "end_time": { 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_time", 2024-04-22T06:47:48.666 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2359", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "end_weekday": { 2024-04-22T06:47:48.667 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_weekday", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.668 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.669 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.670 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.671 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.672 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.673 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "min_score": { 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_score", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.674 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "upmap", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.675 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-compat", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "none", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "read", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap-read" 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Balancer mode", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.676 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_ids": { 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_ids", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.678 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_deviation": { 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_deviation", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:47:48.679 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_optimizations": { 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.680 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm", 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_down_multiplier": { 2024-04-22T06:47:48.681 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_down_multiplier", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_refresh_rate": { 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_refresh_rate", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "20", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.683 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_starting_port": { 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_starting_port", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4721", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.684 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_ptrace": { 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_ptrace", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.685 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_interval": { 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_interval", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.686 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.7", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.687 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm_log_destination": { 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm_log_destination", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "file", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "file,syslog", 2024-04-22T06:47:48.688 INFO:teuthology.orchestra.run.smithi105.stdout: "syslog" 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "cgroups_split": { 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cgroups_split", 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.689 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "config_checks_enabled": { 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_checks_enabled", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.690 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "config_dashboard": { 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_dashboard", 2024-04-22T06:47:48.691 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_alertmanager": { 2024-04-22T06:47:48.692 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_alertmanager", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.693 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_base": { 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_base", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_elasticsearch": { 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "elasticsearch container image", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.695 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_grafana": { 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_grafana", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.696 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_haproxy": { 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_haproxy", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "HAproxy container image", 2024-04-22T06:47:48.697 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_agent": { 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.698 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_collector": { 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.699 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_query": { 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.700 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger query container image", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_keepalived": { 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_keepalived", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:47:48.701 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Keepalived container image", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_loki": { 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_loki", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.702 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Loki container image", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_node_exporter": { 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_node_exporter", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.703 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_nvmeof": { 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_nvmeof", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.704 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Nvme-of container image", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_prometheus": { 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_prometheus", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.705 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_promtail": { 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_promtail", 2024-04-22T06:47:48.706 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Promtail container image", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_samba": { 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_samba", 2024-04-22T06:47:48.707 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.708 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_snmp_gateway": { 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "container_init": { 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_init", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.710 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "daemon_cache_timeout": { 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:47:48.711 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "900", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.712 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "default_registry": { 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_registry", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.713 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "device_cache_timeout": { 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_cache_timeout", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1800", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.714 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "device_enhanced_scan": { 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_enhanced_scan", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.715 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "facts_cache_timeout": { 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "facts_cache_timeout", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.716 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "grafana_dashboards_path": { 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.717 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "host_check_interval": { 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "host_check_interval", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.718 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "hw_monitoring": { 2024-04-22T06:47:48.719 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hw_monitoring", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "inventory_list_all": { 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "inventory_list_all", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.721 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.722 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "log_refresh_metadata": { 2024-04-22T06:47:48.723 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_refresh_metadata", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.724 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.726 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.727 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.728 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.729 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "*", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "max_count_per_host": { 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_count_per_host", 2024-04-22T06:47:48.730 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd_draining_count": { 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_osd_draining_count", 2024-04-22T06:47:48.731 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "migration_current": { 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "migration_current", 2024-04-22T06:47:48.732 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "internal - do not modify", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.733 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "root", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm-package", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "root" 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.734 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "oob_default_addr": { 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "oob_default_addr", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "169.254.1.1", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:47:48.735 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "prometheus_alerts_path": { 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.736 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_insecure": { 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_insecure", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.737 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_password": { 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_password", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.738 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_url": { 2024-04-22T06:47:48.739 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_url", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.740 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_username": { 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_username", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.741 INFO:teuthology.orchestra.run.smithi105.stdout: "secure_monitoring_stack": { 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "service_discovery_port": { 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "service_discovery_port", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8765", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "ssh_config_file": { 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssh_config_file", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.744 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "use_agent": { 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_agent", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.745 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "use_repo_digest": { 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_repo_digest", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.746 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_failed_host_check": { 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.747 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_daemons": { 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.748 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_hosts": { 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:47:48.749 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.750 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crash", 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.751 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.752 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.753 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.754 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.756 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "retain_interval": { 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retain_interval", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "31536000", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:47:48.757 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_recent_interval": { 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_recent_interval", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1209600", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.758 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dashboard", 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:47:48.759 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.760 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:47:48.761 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.762 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.763 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.764 INFO:teuthology.orchestra.run.smithi105.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.765 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.766 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.767 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.768 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.769 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.771 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.772 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.773 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.774 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.775 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_URL": { 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.776 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.777 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.778 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:47:48.779 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.780 INFO:teuthology.orchestra.run.smithi105.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.781 INFO:teuthology.orchestra.run.smithi105.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.782 INFO:teuthology.orchestra.run.smithi105.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "[]", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.783 INFO:teuthology.orchestra.run.smithi105.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "{}", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.785 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.786 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.787 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.787 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:47:48.787 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:47:48.787 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.789 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.790 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.791 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.792 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.793 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.795 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:47:48.796 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:47:48.797 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:47:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.800 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "45", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.801 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.802 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.804 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.805 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.806 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.807 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.808 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "cross_origin_url": { 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cross_origin_url", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "crt_file": { 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crt_file", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.811 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "debug": { 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "debug", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.812 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "jwt_token_ttl": { 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "jwt_token_ttl", 2024-04-22T06:47:48.813 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "28800", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:48.814 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.815 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.816 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.817 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.818 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.820 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "motd": { 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "motd", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "The message of the day", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.821 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.823 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8080", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.824 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.825 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl_server_port": { 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl_server_port", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8443", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.826 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "redirect", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.827 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect" 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:48.828 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "url_prefix": { 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url_prefix", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.829 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "devicehealth", 2024-04-22T06:47:48.830 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_monitoring": { 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_monitoring", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "monitor device health metrics", 2024-04-22T06:47:48.831 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.832 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.833 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.834 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.835 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.836 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "mark_out_threshold": { 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mark_out_threshold", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2419200", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.837 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": { 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_name", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "device_health_metrics", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.838 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "retention_period": { 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retention_period", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15552000", 2024-04-22T06:47:48.839 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_frequency": { 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_frequency", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:48.840 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "self_heal": { 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "self_heal", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.841 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.842 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_threshold": { 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_threshold", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.843 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7257600", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "diskprediction_local", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.845 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.847 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.848 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.850 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "predict_interval": { 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predict_interval", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:48.851 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "predictor_model": { 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predictor_model", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.852 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "prophetstor", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.853 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "influx", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": false, 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "batch_size": { 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "batch_size", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5000", 2024-04-22T06:47:48.855 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "database": { 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "database", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.856 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "ceph", 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.857 INFO:teuthology.orchestra.run.smithi105.stdout: "hostname": { 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hostname", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.858 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "30", 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "5", 2024-04-22T06:47:48.859 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.860 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.861 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.862 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.863 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.864 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.864 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.864 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.864 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.864 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.865 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.866 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.867 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "password": { 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "password", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.868 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "port": { 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "port", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8086", 2024-04-22T06:47:48.869 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server port", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.870 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "false", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "threads": { 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threads", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "32", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "username": { 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "username", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "verify_ssl": { 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "verify_ssl", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "true", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "insights", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "iostat", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "k8sevents", 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_event_retention_days": { 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:47:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "config_check_secs": { 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_check_secs", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "10", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "localpool", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "failure_domain": { 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "failure_domain", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "host", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": { 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_size", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "num_rep": { 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "num_rep", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": { 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_num", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "128", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "prefix": { 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prefix", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "subtree": { 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "subtree", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "rack", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mds_autoscaler", 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirroring", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "nfs", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "fail_fs": { 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "fail_fs", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator": { 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "rook", 2024-04-22T06:47:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "test_orchestrator" 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Orchestrator backend", 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_perf_query", 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_support", 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_autoscaler", 2024-04-22T06:47:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "threshold": { 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threshold", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1.0", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "scaling threshold", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:47:48.944 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "progress", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_pg_recovery_event": { 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "max_completed_events": { 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_completed_events", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "50", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "cache": { 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cache", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "exclude_perf_counters": { 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "exclude_perf_counters", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools": { 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "300", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_interval": { 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_interval", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15.0", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "9283", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "stale_cache_strategy": { 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stale_cache_strategy", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "log", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "default", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "default", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "error" 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_support", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_snap_create": { 2024-04-22T06:47:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:47:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "trash_purge_schedule": { 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "trash_purge_schedule", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "restful", 2024-04-22T06:47:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_auth": { 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_auth", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rgw", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rook", 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "storage_class": { 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "storage_class", 2024-04-22T06:47:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "local", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "selftest", 2024-04-22T06:47:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "roption1": { 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption1", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "roption2": { 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption2", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "xyz", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption1": { 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption1", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption2": { 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption2", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption3": { 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption3", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption4": { 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption4", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption5": { 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption5", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption6": { 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption6", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption7": { 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption7", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "42", 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.014 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "testkey": { 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testkey", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "testlkey": { 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testlkey", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "testnewline": { 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testnewline", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snap_schedule", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_m_granularity": { 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_m_granularity", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.018 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "dump_on_update": { 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dump_on_update", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.019 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.020 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.021 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.022 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.023 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stats", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.025 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.026 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.027 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.028 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.029 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "status", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.030 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.031 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.032 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.033 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.034 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telegraf", 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "address": { 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "address", 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.035 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.036 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.037 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.038 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.039 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.040 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.041 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telemetry", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_basic": { 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_basic", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.042 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_crash": { 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_crash", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.043 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_device": { 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_device", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:47:49.044 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_ident": { 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_ident", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:47:49.045 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_perf": { 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_perf", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:47:49.046 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "contact": { 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "contact", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.047 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "description": { 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "description", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.048 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "device_url": { 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_url", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.049 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.050 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "24", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "8", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.051 INFO:teuthology.orchestra.run.smithi105.stdout: "last_opt_revision": { 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "last_opt_revision", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard": { 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard_description": { 2024-04-22T06:47:49.053 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard_description", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.054 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.056 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.057 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.058 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "organization": { 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "organization", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.059 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "proxy": { 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "proxy", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.060 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "url": { 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.061 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "test_orchestrator", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.062 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.063 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.064 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.065 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.066 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.067 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "volumes", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.068 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.069 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.070 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.071 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.072 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_clones": { 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_clones", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "periodic_async_work": { 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "periodic_async_work", 2024-04-22T06:47:49.073 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Periodically check for async work", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_delay": { 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:47:49.074 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.075 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.076 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "discovery_interval": { 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "discovery_interval", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "100", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.077 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "identifier": { 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "identifier", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.078 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.080 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.081 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.082 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_host": { 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_host", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.084 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_port": { 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_port", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10051", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.085 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_sender": { 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_sender", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.086 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "modules": [ 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "dashboard", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "iostat", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "nfs", 2024-04-22T06:47:49.087 INFO:teuthology.orchestra.run.smithi105.stdout: "restful", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_schedule" 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "available_modules": [ 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "alerts", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.088 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How frequently to reexamine health status", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.089 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.090 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.091 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.092 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_destination": { 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_destination", 2024-04-22T06:47:49.093 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email address to send alerts to", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_from_name": { 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_from_name", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.094 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "Ceph", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Email From: name", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_host": { 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_host", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.095 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP server", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_password": { 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_password", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.096 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Password to authenticate with", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_port": { 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_port", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "465", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.097 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP port", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_sender": { 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_sender", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.098 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SMTP envelope sender", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_ssl": { 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_ssl", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-22T06:47:49.099 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "smtp_user": { 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "smtp_user", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "User to authenticate as", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.100 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "balancer", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "active": { 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "active", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.101 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically balance PGs across cluster", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_time": { 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_time", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0000", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.102 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "beginning time of day to automatically balance", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "begin_weekday": { 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "begin_weekday", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:49.103 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_max_iterations": { 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_max_iterations", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "25", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "250", 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.104 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_metrics": { 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_metrics", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "pgs,objects,bytes", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-22T06:47:49.105 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_compat_step": { 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crush_compat_step", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.5", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0.001", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "0.999", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "aggressiveness of optimization", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-22T06:47:49.106 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "end_time": { 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_time", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2359", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ending time of day to automatically balance", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "end_weekday": { 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "end_weekday", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "0", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "6", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.108 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.109 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.110 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.111 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.112 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "min_score": { 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_score", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.113 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "upmap", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-compat", 2024-04-22T06:47:49.114 INFO:teuthology.orchestra.run.smithi105.stdout: "none", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "read", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap-read" 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Balancer mode", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_ids": { 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_ids", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.115 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.116 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_deviation": { 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_deviation", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.117 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "upmap_max_optimizations": { 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "upmap_max_optimizations", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.118 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_down_multiplier": { 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_down_multiplier", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.119 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_refresh_rate": { 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_refresh_rate", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.120 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "20", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "agent_starting_port": { 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "agent_starting_port", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4721", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.121 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_ptrace": { 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_ptrace", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.122 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_interval": { 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_interval", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.123 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "autotune_memory_target_ratio": { 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "autotune_memory_target_ratio", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0.7", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.124 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm_log_destination": { 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cephadm_log_destination", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "file", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "file,syslog", 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "syslog" 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "cgroups_split": { 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cgroups_split", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.126 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "config_checks_enabled": { 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_checks_enabled", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-22T06:47:49.127 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "config_dashboard": { 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_dashboard", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.128 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_alertmanager": { 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_alertmanager", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.129 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_base": { 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_base", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Container image name, without the tag", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.130 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_elasticsearch": { 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_elasticsearch", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "elasticsearch container image", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_grafana": { 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_grafana", 2024-04-22T06:47:49.131 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_haproxy": { 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_haproxy", 2024-04-22T06:47:49.132 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "HAproxy container image", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_agent": { 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_agent", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.133 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger agent container image", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_collector": { 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_collector", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-22T06:47:49.134 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger collector container image", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_jaeger_query": { 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_jaeger_query", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.135 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Jaeger query container image", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_keepalived": { 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_keepalived", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Keepalived container image", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_loki": { 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_loki", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Loki container image", 2024-04-22T06:47:49.137 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_node_exporter": { 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_node_exporter", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_nvmeof": { 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_nvmeof", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Nvme-of container image", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_prometheus": { 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_prometheus", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Prometheus container image", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_promtail": { 2024-04-22T06:47:49.140 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_promtail", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Promtail container image", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_samba": { 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_samba", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.141 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Samba/SMB container image", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image_snmp_gateway": { 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_image_snmp_gateway", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "SNMP Gateway container image", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "container_init": { 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "container_init", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Run podman/docker with `--init`", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "daemon_cache_timeout": { 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "daemon_cache_timeout", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.144 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "default_cephadm_command_timeout": { 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_cephadm_command_timeout", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "900", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.145 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "default_registry": { 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "default_registry", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "docker.io", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "device_cache_timeout": { 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_cache_timeout", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1800", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache device inventory", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "device_enhanced_scan": { 2024-04-22T06:47:49.147 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_enhanced_scan", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "facts_cache_timeout": { 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "facts_cache_timeout", 2024-04-22T06:47:49.148 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "seconds to cache host facts data", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "grafana_dashboards_path": { 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "grafana_dashboards_path", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.149 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "host_check_interval": { 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "host_check_interval", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.150 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to perform a host check", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "hw_monitoring": { 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hw_monitoring", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.151 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "inventory_list_all": { 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "inventory_list_all", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.152 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.153 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "log_refresh_metadata": { 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_refresh_metadata", 2024-04-22T06:47:49.154 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.156 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.157 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.158 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-22T06:47:49.159 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "*", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "max_count_per_host": { 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_count_per_host", 2024-04-22T06:47:49.160 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of daemons per service per host", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd_draining_count": { 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_osd_draining_count", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.161 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "migration_current": { 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "migration_current", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.162 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "internal - do not modify", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "mode": { 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mode", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "root", 2024-04-22T06:47:49.163 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm-package", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "root" 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "mode for remote execution of cephadm", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "oob_default_addr": { 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "oob_default_addr", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.164 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "169.254.1.1", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "prometheus_alerts_path": { 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus_alerts_path", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-22T06:47:49.165 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_insecure": { 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_insecure", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.166 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_password": { 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_password", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.167 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_url": { 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_url", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.168 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "registry_username": { 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "registry_username", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.169 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "secure_monitoring_stack": { 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "secure_monitoring_stack", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.170 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "service_discovery_port": { 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "service_discovery_port", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8765", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "cephadm service discovery port", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.171 INFO:teuthology.orchestra.run.smithi105.stdout: "ssh_config_file": { 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssh_config_file", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "use_agent": { 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_agent", 2024-04-22T06:47:49.172 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "use_repo_digest": { 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "use_repo_digest", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.173 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_failed_host_check": { 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_failed_host_check", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.174 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if the host check fails", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_daemons": { 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_daemons", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.175 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_on_stray_hosts": { 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_on_stray_hosts", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.176 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crash", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.177 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.178 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.179 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.180 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.181 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "retain_interval": { 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retain_interval", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "31536000", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.182 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_recent_interval": { 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_recent_interval", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1209600", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-22T06:47:49.183 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dashboard", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.184 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_HOST": { 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.185 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.186 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_ENABLED": { 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_ENABLED", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.187 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.188 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "ENABLE_BROWSABLE_API": { 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.190 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-22T06:47:49.191 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-22T06:47:49.192 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.193 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.194 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.197 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.198 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.199 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_PASSWORD": { 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.200 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.201 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_URL": { 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_URL", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.202 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_API_USERNAME": { 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.203 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.205 INFO:teuthology.orchestra.run.smithi105.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-22T06:47:49.206 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.207 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "[]", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "MULTICLUSTER_CONFIG": { 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "{}", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.208 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_HOST": { 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.209 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.210 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.211 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.212 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.213 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.214 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.215 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.216 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.217 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_ENABLED": { 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.218 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-22T06:47:49.219 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.220 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.221 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "45", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.222 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ACCESS_KEY": { 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.223 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "admin", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SECRET_KEY": { 2024-04-22T06:47:49.224 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "RGW_API_SSL_VERIFY": { 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-22T06:47:49.225 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "UNSAFE_TLS_v1_2": { 2024-04-22T06:47:49.226 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-22T06:47:49.227 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-22T06:47:49.229 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "cross_origin_url": { 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cross_origin_url", 2024-04-22T06:47:49.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "crt_file": { 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "crt_file", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.231 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "debug": { 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "debug", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.232 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Enable/disable debug options", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "jwt_token_ttl": { 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "jwt_token_ttl", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "28800", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.233 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.234 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.235 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.236 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.237 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.238 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "motd": { 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "motd", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.239 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "The message of the day", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect_resolve_ip_addr": { 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "redirect_resolve_ip_addr", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.240 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.241 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8080", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.242 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.243 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl_server_port": { 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl_server_port", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8443", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "redirect", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.245 INFO:teuthology.orchestra.run.smithi105.stdout: "redirect" 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.246 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "url_prefix": { 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url_prefix", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.247 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "devicehealth", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_monitoring": { 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_monitoring", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.248 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "monitor device health metrics", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.249 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.250 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.251 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.252 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "mark_out_threshold": { 2024-04-22T06:47:49.253 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mark_out_threshold", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "2419200", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": { 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pool_name", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.254 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "device_health_metrics", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "retention_period": { 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "retention_period", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.255 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15552000", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long to retain device health metrics", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_frequency": { 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_frequency", 2024-04-22T06:47:49.256 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "self_heal": { 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "self_heal", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.257 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:49.258 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how frequently to wake up and check device health", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "warn_threshold": { 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "warn_threshold", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7257600", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "diskprediction_local", 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.260 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.261 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.262 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.263 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.264 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "predict_interval": { 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predict_interval", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "86400", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.265 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "predictor_model": { 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "predictor_model", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "prophetstor", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "600", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "influx", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": false, 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "influxdb python module not found", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "batch_size": { 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "batch_size", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5000", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.268 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "database": { 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "database", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "ceph", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "hostname": { 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "hostname", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server hostname", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.270 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "30", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "5", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.271 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.272 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.273 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.274 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.275 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "password": { 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "password", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "password of InfluxDB server user", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "port": { 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "port", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "8086", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "InfluxDB server port", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.277 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "ssl": { 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ssl", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "false", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "threads": { 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threads", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "32", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.279 INFO:teuthology.orchestra.run.smithi105.stdout: "username": { 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "username", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "username of InfluxDB server user", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "verify_ssl": { 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "verify_ssl", 2024-04-22T06:47:49.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "true", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "insights", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.283 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.284 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.285 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "iostat", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.286 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.287 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.288 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.289 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.290 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "k8sevents", 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_event_retention_days": { 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "ceph_event_retention_days", 2024-04-22T06:47:49.291 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "7", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "config_check_secs": { 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "config_check_secs", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.292 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "10", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.293 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.295 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.296 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "localpool", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "failure_domain": { 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "failure_domain", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "host", 2024-04-22T06:47:49.298 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "failure domain for any created local pool", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.299 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.300 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.301 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": { 2024-04-22T06:47:49.303 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "min_size", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default min_size for any created local pool", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "num_rep": { 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "num_rep", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.304 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default replica count for any created local pool", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": { 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_num", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "128", 2024-04-22T06:47:49.305 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "default pg_num for any created local pool", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "prefix": { 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prefix", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.306 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "name prefix for any created local pool", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "subtree": { 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "subtree", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "rack", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-22T06:47:49.307 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mds_autoscaler", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.308 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.310 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.311 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.312 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirroring", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.313 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.316 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "nfs", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.318 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.319 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.320 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.321 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.322 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "fail_fs": { 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "fail_fs", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.323 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.324 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.325 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.326 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.327 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.328 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator": { 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "orchestrator", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "rook", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "test_orchestrator" 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Orchestrator backend", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.329 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_perf_query", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.330 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.331 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.332 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.334 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "osd_support", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.335 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.336 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.337 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.338 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.339 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "pg_autoscaler", 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.340 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.341 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.342 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.343 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:49.344 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "threshold": { 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "threshold", 2024-04-22T06:47:49.345 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "3.0", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1.0", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "scaling threshold", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "progress", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_pg_recovery_event": { 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_pg_recovery_event", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.348 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.349 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.350 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.351 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.352 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "max_completed_events": { 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_completed_events", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "50", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.353 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "number of past completed events to remember", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "sleep_interval": { 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "sleep_interval", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "5", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "how long the module is going to sleep", 2024-04-22T06:47:49.354 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "prometheus", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "cache": { 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "cache", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.355 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "exclude_perf_counters": { 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "exclude_perf_counters", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.356 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.357 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.358 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.359 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.360 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools": { 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.361 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "300", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.362 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "scrape_interval": { 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "scrape_interval", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15.0", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.363 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "::", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.364 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "9283", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-22T06:47:49.365 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "stale_cache_strategy": { 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stale_cache_strategy", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "log", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.366 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_behaviour": { 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_behaviour", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "default", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "default", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "error" 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.367 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "standby_error_status_code": { 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "standby_error_status_code", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "500", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "400", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "599", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.368 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_support", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.369 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.370 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.371 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.372 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_snap_create": { 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_snap_create", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.373 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "mirror_snapshot_schedule": { 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mirror_snapshot_schedule", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.374 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "trash_purge_schedule": { 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "trash_purge_schedule", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.375 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "restful", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_auth": { 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enable_auth", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.376 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "key_file": { 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "key_file", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.377 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.378 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.379 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.381 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.382 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "server_addr": { 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_addr", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "server_port": { 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "server_port", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rgw", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.385 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.386 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.387 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.388 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.389 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rook", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.390 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.391 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.392 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.393 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "storage_class": { 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "storage_class", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.394 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "local", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "selftest", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.395 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.396 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.397 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.398 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.399 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "roption1": { 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption1", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.400 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "roption2": { 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "roption2", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "xyz", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.401 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption1": { 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption1", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption2": { 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption2", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.403 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption3": { 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption3", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "float", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption4": { 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption4", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption5": { 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption5", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.405 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption6": { 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption6", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.406 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "rwoption7": { 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rwoption7", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "1", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "42", 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.407 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "testkey": { 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testkey", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.408 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "testlkey": { 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testlkey", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "testnewline": { 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "testnewline", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.410 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snap_schedule", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_m_granularity": { 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "allow_m_granularity", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.411 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "allow minute scheduled snapshots", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "dump_on_update": { 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "dump_on_update", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "dump database to debug log on update", 2024-04-22T06:47:49.412 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.413 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.414 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.415 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "stats", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.418 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.419 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.420 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "status", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.423 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.424 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.425 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.426 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telegraf", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.427 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "address": { 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "address", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.428 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "15", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.431 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.432 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.433 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "telemetry", 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_basic": { 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_basic", 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.434 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_crash": { 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_crash", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.435 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_device": { 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_device", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_ident": { 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_ident", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "channel_perf": { 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "channel_perf", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "contact": { 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "contact", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "description": { 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "description", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "device_url": { 2024-04-22T06:47:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "device_url", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "enabled": { 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "enabled", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "24", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "8", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "last_opt_revision": { 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "last_opt_revision", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "1", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard": { 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "leaderboard_description": { 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "leaderboard_description", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "organization": { 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "organization", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "proxy": { 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "proxy", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "url": { 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "url", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "test_orchestrator", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "volumes", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "max_concurrent_clones": { 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "max_concurrent_clones", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "4", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "periodic_async_work": { 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "periodic_async_work", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Periodically check for async work", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_delay": { 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_delay", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "0", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "snapshot_clone_no_wait": { 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snapshot_clone_no_wait", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "True", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix", 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "can_run": true, 2024-04-22T06:47:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "error_string": "", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "module_options": { 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "discovery_interval": { 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "discovery_interval", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "uint", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "100", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "identifier": { 2024-04-22T06:47:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "identifier", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "interval": { 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "interval", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "secs", 2024-04-22T06:47:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "60", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "log_level": { 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_level", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster": { 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_cluster_level": { 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_cluster_level", 2024-04-22T06:47:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "info", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [ 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "critical", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "debug", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "error", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "info", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "warning" 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": { 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "log_to_file", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "bool", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "False", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_host": { 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_host", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_port": { 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_port", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "int", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "10051", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "zabbix_sender": { 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "zabbix_sender", 2024-04-22T06:47:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "str", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "level": "advanced", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 0, 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "min": "", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "max": "", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "enum_allowed": [], 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "desc": "", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "long_desc": "", 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "tags": [], 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "see_also": [] 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "services": { 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "dashboard": "https://172.21.15.105:8443/" 2024-04-22T06:47:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "always_on_modules": { 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "octopus": [ 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "balancer", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "crash", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "devicehealth", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscaler", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "progress", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_support", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "status", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "telemetry", 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "volumes" 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "pacific": [ 2024-04-22T06:47:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "balancer", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "crash", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "devicehealth", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscaler", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "progress", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_support", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "status", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "telemetry", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "volumes" 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "quincy": [ 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "balancer", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "crash", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "devicehealth", 2024-04-22T06:47:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscaler", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "progress", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_support", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "status", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "telemetry", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "volumes" 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "reef": [ 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "balancer", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "crash", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "devicehealth", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscaler", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "progress", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_support", 2024-04-22T06:47:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "status", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "telemetry", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "volumes" 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "squid": [ 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "balancer", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "crash", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "devicehealth", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "orchestrator", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscaler", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "progress", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_support", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "status", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "telemetry", 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "volumes" 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "last_failure_osd_epoch": 92, 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "active_clients": [ 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "devicehealth", 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 2561448807 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "libcephsqlite", 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 152598748 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "rbd_support", 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 2969991774 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snap_schedule", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 4191402336 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "snap_schedule", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 3954931468 2024-04-22T06:47:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "volumes", 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:0", 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 2031819216 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-22T06:47:49.484 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:47:49.485 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:48 smithi177 ceph-mon[31346]: pgmap v328: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2659901128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/783565400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:48 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1941323215' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:48 smithi112 ceph-mon[31236]: pgmap v328: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2659901128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/783565400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.486 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:48 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1941323215' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 msgr2=0x7f28080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 0x7f28080a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2818005fd0 tx=0x7f2818009680 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 msgr2=0x7f281c198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f280c00d950 tx=0x7f280c00de10 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 shutdown_connections 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f28080a3d20 0x7f28080a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f281c10c520 0x7f281c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f281c103b00 0x7f281c198c90 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 --2- 172.21.15.105:0/1941323215 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f281c1031e0 0x7f281c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 >> 172.21.15.105:0/1941323215 conn(0x7f281c077f50 msgr2=0x7f281c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:47:49.487 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 shutdown_connections 2024-04-22T06:47:49.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:47:48.226+0000 7f2823dca640 1 -- 172.21.15.105:0/1941323215 wait complete. 2024-04-22T06:47:49.494 DEBUG:tasks.check_counter:Getting stats from a 2024-04-22T06:47:49.494 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-22T06:47:49.494 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:48 smithi105 ceph-mon[27916]: pgmap v328: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:47:49.495 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2659901128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.495 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/783565400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:47:49.495 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:48 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1941323215' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-22T06:47:49.741 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:47:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:50 smithi105 ceph-mon[27916]: pgmap v329: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:50.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:50 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:50.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:50 smithi112 ceph-mon[31236]: pgmap v329: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:50.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:50 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:50.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:50 smithi177 ceph-mon[31346]: pgmap v329: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:50.739 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:50 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 450, 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 450, 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 152793, 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 142277, 2024-04-22T06:47:51.527 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 6, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 2, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.056994270, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.023445167, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.012471336, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.014820340, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 449, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.028847209, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000064247 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:51.528 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 893, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 986, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 228596, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 532945, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 4, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.069216131, 2024-04-22T06:47:51.529 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.022589469, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.017694677, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.023744193, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 984, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.033052977, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000033590 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:47:51.530 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 249232 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 137, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 136, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 199236, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 77653, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.020982550, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.009174873, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.008154902, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.002820360, 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:51.531 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 135, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.009181343, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000068009 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 197888, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 76688 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-22T06:47:51.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2228, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.058958162, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000026462 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.533 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 0, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 0, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 0, 2024-04-22T06:47:51.534 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 0, 2024-04-22T06:47:51.535 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 1, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-04-22T06:47:51.536 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:47:51.537 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 0 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-04-22T06:47:51.538 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-04-22T06:47:51.539 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 0, 2024-04-22T06:47:51.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 7, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 2, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 2, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 4194304, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 4196124, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 4195983, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.541 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 7 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 3, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 3, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 0, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 1, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 1, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 0, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 0, 2024-04-22T06:47:51.542 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 49824, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 215416 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 0, 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-04-22T06:47:51.543 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-04-22T06:47:51.544 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.545 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.546 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-04-22T06:47:51.547 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.548 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-04-22T06:47:51.553 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 0, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 2, 2024-04-22T06:47:51.554 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 2, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 2, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:47:51.555 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:47:51.556 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 197319, 2024-04-22T06:47:51.557 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 52, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9872, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 249, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-22T06:47:51.558 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 8512, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 13, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 1341, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 21, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 0, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 1341, 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-04-22T06:47:51.559 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1341, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.651987122, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000486194 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1341, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1342 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 1341, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 1341, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 0, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 890, 2024-04-22T06:47:51.560 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 449, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 1, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-04-22T06:47:51.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-04-22T06:47:51.562 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 112, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 8, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 6, 2024-04-22T06:47:51.563 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 6, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 2, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 0 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-04-22T06:47:51.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 1476, 2024-04-22T06:47:51.565 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 437914, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 1476, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 1476, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 437914, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:47:51.566 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 1341, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 1825, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 1341, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1825, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.567 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 1341, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 1341, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 1341, 2024-04-22T06:47:51.568 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1341, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:51.569 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x560f5f6ce780": { 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:51.570 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:51.572 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:47:52.313 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-22T06:47:52.313 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-22T06:47:52.313 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-22T06:47:52.313 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:47:52.313 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:47:52.313 DEBUG:tasks.check_counter:Getting stats from d 2024-04-22T06:47:52.313 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: pgmap v330: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:52 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: pgmap v330: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:52 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: pgmap v330: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:52.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:52 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:53.073 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:47:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:53 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:47:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:53 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:47:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:53 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:47:53.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:53 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:47:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:53 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:47:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:53 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:47:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:53 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:47:53.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:53 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:47:53.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:53 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi177 to 176.1M 2024-04-22T06:47:53.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:53 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi177 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-22T06:47:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:53 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-04-22T06:47:53.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:53 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi112 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-04-22T06:47:54.739 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:54 smithi105 ceph-mon[27916]: pgmap v331: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 216, 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 205, 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1588240, 2024-04-22T06:47:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 125984, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.042439725, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.014164030, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.032654840, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.004238898, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 204, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.010613928, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000052029 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1588240, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 125984 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 6, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 159, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 3312, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 184192, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.013665027, 2024-04-22T06:47:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.009510711, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.002080951, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 157, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.018677781, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000118966 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 184192 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000335985, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000265450, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:47:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-22T06:47:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:47:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:47:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 150025, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 32, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-22T06:47:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:47:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 218, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1557789, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 218, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 218, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1557789, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:54.751 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:47:54.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:54 smithi112 ceph-mon[31236]: pgmap v331: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:54.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:54 smithi177 ceph-mon[31346]: pgmap v331: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:47:55.406 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-22T06:47:55.406 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-22T06:47:55.406 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-22T06:47:55.406 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:47:55.406 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-22T06:47:55.406 DEBUG:tasks.check_counter:Getting stats from g 2024-04-22T06:47:55.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-22T06:47:55.664 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:47:56.793 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: pgmap v332: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:56.793 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.793 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:56.794 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:56 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: pgmap v332: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:56.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:56 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: pgmap v332: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T06:47:56.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:56 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 72, 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 74, 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 34770, 2024-04-22T06:47:57.407 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 31472, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 14, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 18446744073709551615, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.011962955, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.002917901, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.005018612, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.001810387, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 72, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.007658131, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000106362 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 7, 2024-04-22T06:47:57.408 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000001197, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000171 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 276, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 495, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 101081, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 367712, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 12, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 4, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.046252417, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.020491379, 2024-04-22T06:47:57.409 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.017259351, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.005552696, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 494, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.032540721, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000065871 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 415, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000176114, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000424 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 271184 2024-04-22T06:47:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 186, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 174, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 283101, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 102994, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 8, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.025406302, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.010594188, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.010240119, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.003417300, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 173, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.012809628, 2024-04-22T06:47:57.411 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000074044 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 8, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000001590, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000198 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 262208, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 89376 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.412 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 301, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.009529670, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000031660 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000078851, 2024-04-22T06:47:57.413 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000019712 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 0, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 0, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 0, 2024-04-22T06:47:57.414 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 11, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 10, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 10, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 7, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 3, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 10, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 0, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 1, 2024-04-22T06:47:57.415 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-04-22T06:47:57.416 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 4, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 234, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 3, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:47:57.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 0 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 1, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-04-22T06:47:57.418 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:47:57.419 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 7, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 0, 2024-04-22T06:47:57.420 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 2, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 2, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 4194304, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 4196124, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 4195983, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 7 2024-04-22T06:47:57.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 13, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 14, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 1, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 11, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 12, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 1, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 10, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 10, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 0, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 0, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 0, 2024-04-22T06:47:57.422 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 50972, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 239992 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.423 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.424 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.426 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.427 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.428 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.430 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 0, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 2, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 2, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 2, 2024-04-22T06:47:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:47:57.435 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:47:57.436 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 263075, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 95, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-22T06:47:57.437 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9872, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 249, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 52984, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 175, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-22T06:47:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 196, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 421, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 0, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 196, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 196, 2024-04-22T06:47:57.439 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.249995053, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001275484 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 196, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 240 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 196, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 189, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 7, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-04-22T06:47:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 107, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 58, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 1, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 5, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 22, 2024-04-22T06:47:57.441 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-04-22T06:47:57.442 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-04-22T06:47:57.443 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 112, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 12, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 9, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 9, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 1, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 46, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 0 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-04-22T06:47:57.444 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:57.445 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 530, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 359624, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 530, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 530, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 359624, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 196, 2024-04-22T06:47:57.447 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 1993, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 196, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1993, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:57.448 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 196, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 196, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 196, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 196, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.449 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-04-22T06:47:57.450 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x55aa15fda780": { 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-04-22T06:47:57.451 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:47:57.452 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:47:57.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:57 smithi105 ceph-mon[27916]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:47:57.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:57 smithi105 ceph-mon[27916]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:47:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:57 smithi112 ceph-mon[31236]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:47:57.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:57 smithi112 ceph-mon[31236]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:47:57.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:57 smithi177 ceph-mon[31346]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-04-22T06:47:57.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:57 smithi177 ceph-mon[31346]: Unable to set osd_memory_target on smithi105 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-04-22T06:47:58.191 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-22T06:47:58.191 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-22T06:47:58.191 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=3 2024-04-22T06:47:58.191 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:47:58.191 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:47:58.191 DEBUG:tasks.check_counter:Getting stats from j 2024-04-22T06:47:58.191 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-22T06:47:58.438 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T06:47:58.897 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:47:58 smithi105 ceph-mon[27916]: pgmap v333: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:58.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:47:58 smithi112 ceph-mon[31236]: pgmap v333: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:47:58.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:47:58 smithi177 ceph-mon[31346]: pgmap v333: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-04-22T06:48:00.044 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000991406, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000181161, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:48:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 218, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 207, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1588784, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 127264, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-04-22T06:48:00.046 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.044135806, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.014462436, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.034576393, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.004223667, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 206, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.011246739, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000054595 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:48:00.047 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1588784, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 127264 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 6, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 160, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 3312, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 184368, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:00.048 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.013464795, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008821042, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.002374435, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 158, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.021668241, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000137140 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:48:00.049 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 184368 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-04-22T06:48:00.050 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:00.052 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148955, 2024-04-22T06:48:00.053 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 30, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:00.054 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 220, 2024-04-22T06:48:00.055 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1558049, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 220, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 220, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1558049, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:48:00.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-22T06:48:00.057 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-22T06:48:00.798 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-22T06:48:00.798 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-22T06:48:00.798 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-22T06:48:00.799 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:48:00.799 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-22T06:48:00.799 DEBUG:tasks.check_counter:Getting stats from b 2024-04-22T06:48:00.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-22T06:48:00.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:00 smithi105 ceph-mon[27916]: pgmap v334: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:48:00.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:00 smithi112 ceph-mon[31236]: pgmap v334: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:48:00.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:00 smithi177 ceph-mon[31346]: pgmap v334: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-22T06:48:01.047 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:48:02.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:02 smithi112 ceph-mon[31236]: pgmap v335: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:48:02.795 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-22T06:48:02.795 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 1537, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 1193, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 470689, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 1612058, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 39, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.152039724, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.045291573, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.058503481, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.036996633, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1141, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.483780230, 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000423996 2024-04-22T06:48:02.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1680, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000332996, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000198 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 44672, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 6096 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 2441, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 1232, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 1025005, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 935784, 2024-04-22T06:48:02.797 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 41, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.293120825, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.072520037, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.114907654, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.100461139, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1191, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.480939105, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000403811 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2969, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000671054, 2024-04-22T06:48:02.798 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000226 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 323760, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 226592 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 1220, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 1074, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 2284618, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 986649, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 38, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 8, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.155717233, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.043326827, 2024-04-22T06:48:02.799 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.084068426, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.036766737, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1030, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.470200072, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000456504 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 915, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000207540, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000226 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 1935696, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 484960 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.800 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 580, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.117454892, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000202508 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 89, 2024-04-22T06:48:02.801 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.002711052, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000030461 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 184, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 184, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 184, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.166951654, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000907345 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 0, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 0, 2024-04-22T06:48:02.802 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 0, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 60, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 0, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 39, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 7, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 32, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 29, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 0, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 3, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 4, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 2, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 239, 2024-04-22T06:48:02.803 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 230, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 35, 2024-04-22T06:48:02.804 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 6, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 35, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 23, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 3, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 20, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 12, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 1, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 32, 2024-04-22T06:48:02.805 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 43, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 97, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 62, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 4, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 27 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 0, 2024-04-22T06:48:02.806 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 19, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 2, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 6, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 4, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 3, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 41, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 3, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:02.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 224, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 141, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 141, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 83, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 11, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 13, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 12, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 12, 2024-04-22T06:48:02.808 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 1, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 1, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 1, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4860414, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 5099642, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 0, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 183, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.477653753, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002610129 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 0 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-04-22T06:48:02.809 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 29, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 36, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 7, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 20, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 20, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 0, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 39, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 43, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 4, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 4, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 27, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 23, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 55812, 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 231800 2024-04-22T06:48:02.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 235, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 18, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 184, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 33, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 31, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.005066643, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000633330 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 111, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.021682408, 2024-04-22T06:48:02.811 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000195337 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:02.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.001941432, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000242679 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000141161, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000141161 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.813 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 11, 2024-04-22T06:48:02.814 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.002120694, 2024-04-22T06:48:02.815 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000192790 2024-04-22T06:48:02.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.815 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.012674551, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001584318 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 11, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.060380325, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.005489120 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.816 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000205065, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000205065 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 3, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.003633326, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001211108 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2, 2024-04-22T06:48:02.817 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.001956234, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000978117 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-04-22T06:48:02.818 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.008714383, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.004357191 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 5, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.024185120, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.004837024 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.819 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.001000738, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001000738 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.018608294, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002326036 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 4, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.004641280, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.001160320 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 2, 2024-04-22T06:48:02.822 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 5, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 3, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 2, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 27, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 13, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 612, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:02.823 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:02.824 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:02.825 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 327749, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 120, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 249, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:02.826 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 137198, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 1189, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 996, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 1022946, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 37, 2024-04-22T06:48:02.827 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 996, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 996, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 34.074105458, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.034210949 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 996, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 1168 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 996, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 43, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 953, 2024-04-22T06:48:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 40, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 28, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 5, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 185, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 60, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 579, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 38, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 56, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-04-22T06:48:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-04-22T06:48:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 112, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 24, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 12, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 12, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 163, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 14 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 1, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-04-22T06:48:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 61 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 64, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 3, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 67, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 4, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 5194, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 3244433, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 5194, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 5194, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 3244433, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 996, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 1023382, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 996, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 1023382, 2024-04-22T06:48:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 996, 2024-04-22T06:48:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 996, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 996, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 996, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 4, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 436, 2024-04-22T06:48:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 4, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 2, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 436, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x5597415d2500": { 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 84, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 224, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 905338, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 224, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 183, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 905254, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-22T06:48:02.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:02 smithi105 ceph-mon[27916]: pgmap v335: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:48:02.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:02 smithi177 ceph-mon[31346]: pgmap v335: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-22T06:48:03.526 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-22T06:48:03.526 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-22T06:48:03.526 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-22T06:48:03.526 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-04-22T06:48:03.526 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-04-22T06:48:03.526 DEBUG:tasks.check_counter:Getting stats from e 2024-04-22T06:48:03.526 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-22T06:48:03.777 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:48:04.838 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:04 smithi112 ceph-mon[31236]: pgmap v336: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:04.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:04 smithi105 ceph-mon[27916]: pgmap v336: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:04.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:04 smithi177 ceph-mon[31346]: pgmap v336: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:05.384 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 75, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 79, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 26967, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 24108, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 11, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.008645079, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.001641978, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.003725435, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.001619786, 2024-04-22T06:48:05.385 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 74, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.007614077, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000102892 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:48:05.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 237, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 225, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 289468, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 119747, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 7, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.030789768, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.011264140, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.013431843, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.005565742, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 224, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.015825667, 2024-04-22T06:48:05.387 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000070650 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000001919, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000239 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 262752, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 90656 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 232, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 447, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 112565, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 361419, 2024-04-22T06:48:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 8, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.043509216, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.021207619, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.014631141, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.004409553, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 449, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.027115088, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000060389 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 440, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000114982, 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000261 2024-04-22T06:48:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 272528 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 291, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.010607597, 2024-04-22T06:48:05.390 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000036452 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 4, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000084256, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000021064 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 0, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 0, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.391 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 11, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 10, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 10, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 7, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 3, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 10, 2024-04-22T06:48:05.392 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 1, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 0, 2024-04-22T06:48:05.393 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 3, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:05.394 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 0 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 1, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:05.395 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:05.396 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 5, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-04-22T06:48:05.397 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 2, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 2, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4194304, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 4195540, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 4195399, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.398 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 5 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 13, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 14, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 1, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 11, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 12, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 1, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 10, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 10, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 0, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 0, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 0, 2024-04-22T06:48:05.399 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 50244, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 239992 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.400 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-04-22T06:48:05.401 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.402 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.404 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.405 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.406 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.407 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.409 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.410 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 1, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 1, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 1, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:05.411 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:05.412 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:05.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 230227, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 76, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 249, 2024-04-22T06:48:05.414 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 52984, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 175, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-04-22T06:48:05.415 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 193, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 421, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 0, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 193, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 193, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.427988766, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002217558 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 193, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 237 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 193, 2024-04-22T06:48:05.416 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 186, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 7, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 105, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 57, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 1, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 5, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:05.417 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 22, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-04-22T06:48:05.418 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 112, 2024-04-22T06:48:05.419 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 12, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 9, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 9, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 1, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 46, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 0 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.420 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 0 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 540, 2024-04-22T06:48:05.421 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 368668, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 540, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 540, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 368668, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:05.422 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 193, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 1409, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 193, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 1409, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:05.423 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 193, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 193, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 193, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 193, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:05.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.425 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x5606443d0780": { 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:05.426 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:05.427 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:05.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:05.427 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-22T06:48:06.141 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-22T06:48:06.141 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-22T06:48:06.141 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=3 2024-04-22T06:48:06.141 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:06.142 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:06.142 DEBUG:tasks.check_counter:Getting stats from h 2024-04-22T06:48:06.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-22T06:48:06.381 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:48:06.803 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:06 smithi112 ceph-mon[31236]: pgmap v337: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:06.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:06 smithi105 ceph-mon[27916]: pgmap v337: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:06.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:06 smithi177 ceph-mon[31346]: pgmap v337: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 601, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 600, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 321502, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 226482, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 5, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 1, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.058916800, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.020929775, 2024-04-22T06:48:07.848 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.017810031, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.014717845, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 599, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.020947687, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000034971 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 198976, 2024-04-22T06:48:07.849 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 79248 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 468, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 562, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 117072, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 400674, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 5, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.047310877, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.015500064, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.011209114, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.016295481, 2024-04-22T06:48:07.850 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 561, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.019921016, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000035509 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 253440 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 466, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 468, 2024-04-22T06:48:07.851 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 156903, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 148257, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 5, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.056450610, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.021588195, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.012917730, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.016077336, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 466, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.030162847, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000064727 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.852 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 2263, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.064920263, 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000028687 2024-04-22T06:48:07.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 0, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 0, 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-04-22T06:48:07.854 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 0, 2024-04-22T06:48:07.855 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 1, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 0, 2024-04-22T06:48:07.856 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:07.857 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 0 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:07.858 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:07.859 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 5, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-04-22T06:48:07.860 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 2, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 2, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4194304, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 4195540, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 4195399, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.861 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 5 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 3, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 3, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 1, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 1, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 0, 2024-04-22T06:48:07.862 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 52152, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 215416 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-04-22T06:48:07.863 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.864 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-04-22T06:48:07.865 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.866 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.868 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.869 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.870 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.871 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.872 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 1, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 1, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 1, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 0, 2024-04-22T06:48:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 198350, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 53, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-04-22T06:48:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 249, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 8512, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 13, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-04-22T06:48:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 1392, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 21, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 0, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 1392, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1392, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.737980621, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000530158 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1392, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 1393 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 1392, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 1392, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 924, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 466, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 0, 2024-04-22T06:48:07.880 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 1, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-04-22T06:48:07.881 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-04-22T06:48:07.882 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 112, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 8, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 6, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 6, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 0, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 2, 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 0 2024-04-22T06:48:07.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 0 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:07.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 1531, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 447608, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 1531, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1531, 2024-04-22T06:48:07.885 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 447608, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:07.886 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 1392, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 1241, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1392, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 1241, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:07.887 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 1392, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 1392, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1392, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 1392, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.889 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x55983db50780": { 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-04-22T06:48:07.890 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:07.891 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-22T06:48:08.613 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-22T06:48:08.613 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-22T06:48:08.614 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-22T06:48:08.614 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:08.614 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:08.614 DEBUG:tasks.check_counter:Getting stats from k 2024-04-22T06:48:08.614 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-22T06:48:08.851 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.b/config 2024-04-22T06:48:08.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:08 smithi105 ceph-mon[27916]: pgmap v338: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:08.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:08 smithi112 ceph-mon[31236]: pgmap v338: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:08.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:08 smithi177 ceph-mon[31346]: pgmap v338: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-22T06:48:10.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 187, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 187, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 67854, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 67142, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 11, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.017820767, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.004719878, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.006209716, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.004397515, 2024-04-22T06:48:10.312 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 186, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.014545247, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000078200 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 8, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000001851, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000231 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:10.313 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 194, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 180, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 292426, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 114156, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 7, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 5, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.026888601, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.010355339, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.011641366, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.003827307, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 181, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.016571638, 2024-04-22T06:48:10.314 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000091556 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 15, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000003008, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000200 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 263184, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 91664 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_messages": 177, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages": 416, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_bytes": 67019, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_bytes": 409543, 2024-04-22T06:48:10.315 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_created_connections": 10, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_active_connections": 6, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_total_time": 0.043737837, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_send_time": 0.021567304, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_recv_time": 0.016096241, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_running_fast_dispatch_time": 0.002906705, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 412, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.024855110, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000060327 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 458, 2024-04-22T06:48:10.316 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000120526, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000263 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "msgr_send_encrypted_bytes": 274320 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "cct": { 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_workers": 1, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "unhealthy_workers": 0 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-MDSRank": { 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 304, 2024-04-22T06:48:10.317 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.010089559, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000033189 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "queue_len": 0, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "complete_latency": { 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 4, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000121174, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000030293 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "request": 0, 2024-04-22T06:48:10.318 INFO:teuthology.orchestra.run.smithi112.stdout: "reply": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "reply_latency": { 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_reply": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "forward": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_complete": 11, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_fetch_keys": 10, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_commit": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_split": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_merge": 0, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes": 10, 2024-04-22T06:48:10.319 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_top": 7, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_bottom": 3, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_pinned": 10, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_expired": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "subtrees": 1, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_hit": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_forward": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_discover": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "traverse_lock": 0, 2024-04-22T06:48:10.320 INFO:teuthology.orchestra.run.smithi112.stdout: "load_cent": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "q": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "exported": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "exported_inodes": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "imported": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "imported_inodes": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rfiles": 4, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rbytes": 234, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "root_rsnaps": 3, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:10.321 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: "process_request_cap_release": 0 2024-04-22T06:48:10.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_cache": { 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_try_discover": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_send_discover": 1, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_created": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_enqueued": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "strays_migrated": 0, 2024-04-22T06:48:10.323 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_started": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "recovery_completed": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_flush": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "ireq_inodestats": 0 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_log": { 2024-04-22T06:48:10.324 INFO:teuthology.orchestra.run.smithi112.stdout: "evlrg": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "evadd": 1, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "evex": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "evtrm": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "ev": 10, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "evexg": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "evexd": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segadd": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segex": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segtrm": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "seg": 2, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segmjr": 2, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segexg": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "segexd": 0, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "expos": 4194304, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "wrpos": 4196716, 2024-04-22T06:48:10.325 INFO:teuthology.orchestra.run.smithi112.stdout: "rdpos": 4196484, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "jlat": { 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 1, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.002409886, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002409886 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "replayed": 9 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_mem": { 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "ino": 13, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "ino+": 14, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "ino-": 1, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "dir": 11, 2024-04-22T06:48:10.326 INFO:teuthology.orchestra.run.smithi112.stdout: "dir+": 12, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dir-": 1, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dn": 10, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dn+": 10, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dn-": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "cap": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "cap+": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "cap-": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "rss": 50572, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "heap": 239992 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_server": { 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_request": 0, 2024-04-22T06:48:10.327 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_client_session": 6, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "handle_peer_request": 0, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "req_create_latency": { 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getattr_latency": { 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.328 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "req_link_latency": { 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookup_latency": { 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupino_latency": { 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupname_latency": { 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:10.330 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "req_lssnap_latency": { 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mkdir_latency": { 2024-04-22T06:48:10.331 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mknod_latency": { 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.332 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "req_mksnap_latency": { 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "req_open_latency": { 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "req_readdir_latency": { 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.333 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rename_latency": { 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.334 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmdir_latency": { 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.335 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setattr_latency": { 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setlayout_latency": { 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "req_setxattr_latency": { 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "req_symlink_latency": { 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.337 INFO:teuthology.orchestra.run.smithi112.stdout: "req_unlink_latency": { 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_sessions": { 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "session_count": 1, 2024-04-22T06:48:10.338 INFO:teuthology.orchestra.run.smithi112.stdout: "session_add": 3, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "session_remove": 2, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_open": 3, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "sessions_stale": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_load": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "average_load": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "avg_session_uptime": 4183, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "mempool": { 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:10.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:10.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_bytes": 262311, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_anon_items": 89, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:10.341 INFO:teuthology.orchestra.run.smithi112.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_bytes": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_items": 249, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pgmap_items": 0, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_bytes": 52984, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_co_items": 175, 2024-04-22T06:48:10.342 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_1_items": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "unittest_2_items": 0 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "objecter": { 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_active": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_laggy": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send": 198, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_send_bytes": 627, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_resend": 0, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_reply": 198, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "op_latency": { 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 198, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.418989000, 2024-04-22T06:48:10.343 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.002116106 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op_inflight": 0, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "oplen_avg": { 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 198, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 242 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op": 198, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op_r": 189, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op_w": 9, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op_rmw": 0, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "op_pg": 0, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_stat": 107, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_create": 0, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_read": 58, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_write": 1, 2024-04-22T06:48:10.344 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writefull": 1, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_writesame": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_append": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_zero": 1, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_truncate": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_delete": 5, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_mapext": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_getxattr": 22, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_call": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_watch": 0, 2024-04-22T06:48:10.345 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_notify": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "osdop_other": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_active": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_send": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_resend": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "linger_ping": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_active": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_send": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolop_resend": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_active": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_send": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "poolstat_resend": 0, 2024-04-22T06:48:10.346 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_active": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_send": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "statfs_resend": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "command_active": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "command_send": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "command_resend": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "map_epoch": 112, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "map_full": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "map_inc": 12, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_sessions": 10, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_open": 10, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_session_close": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_laggy": 0, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_wr": 1, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_rd": 46, 2024-04-22T06:48:10.347 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_del": 0 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "oft": { 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_objs": 0, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_updates": 0, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "omap_total_removes": 0 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "osd-slow-ops": { 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "slow_ops_count": 0 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "purge_queue": { 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:10.348 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_executed": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:10.349 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 554, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 365613, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 554, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 554, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 365613, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.350 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 104857600, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:10.351 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 198, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 2675, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 198, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 2675, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.352 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 1024, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 198, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 198, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 198, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 198, 2024-04-22T06:48:10.353 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 0, 2024-04-22T06:48:10.354 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-04-22T06:48:10.355 INFO:teuthology.orchestra.run.smithi112.stdout: "throttle-write_buf_throttle-0x56092d1a0780": { 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 0, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "max": 3758096384, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "get_started": 0, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "get": 1, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "get_sum": 116, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "get_or_fail_success": 1, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "take": 0, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "take_sum": 0, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "put": 1, 2024-04-22T06:48:10.356 INFO:teuthology.orchestra.run.smithi112.stdout: "put_sum": 116, 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "wait": { 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgcount": 0, 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "sum": 0.000000000, 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: "avgtime": 0.000000000 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-04-22T06:48:10.357 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-04-22T06:48:10.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:10 smithi105 ceph-mon[27916]: pgmap v339: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:10.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:10 smithi112 ceph-mon[31236]: pgmap v339: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:10.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:10 smithi177 ceph-mon[31346]: pgmap v339: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-22T06:48:10.993 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-22T06:48:10.993 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-22T06:48:10.993 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=3 2024-04-22T06:48:10.993 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:10.993 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:10.993 DEBUG:tasks.check_counter:Getting stats from c 2024-04-22T06:48:10.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-22T06:48:11.243 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:48:11.686 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:11 smithi177 ceph-mon[31346]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:48:11.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:11 smithi105 ceph-mon[27916]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:48:11.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:11 smithi112 ceph-mon[31236]: osdmap e113: 12 total, 12 up, 12 in 2024-04-22T06:48:12.689 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:12 smithi177 ceph-mon[31346]: pgmap v341: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:48:12.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:12 smithi105 ceph-mon[27916]: pgmap v341: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 941, 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 941, 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 244556, 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 298472, 2024-04-22T06:48:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.071188284, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.018021658, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.019120255, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.028392427, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 940, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.035500078, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000037766 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 3, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 100, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 674, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 257041, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 5, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.006831083, 2024-04-22T06:48:12.933 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.004099992, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001590336, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000051927, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 98, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.008103767, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000082691 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.934 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 256400 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 608, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 607, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 357858, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 228894, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.076675360, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.026317966, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.022496361, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.021057773, 2024-04-22T06:48:12.935 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 606, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.036269382, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000059850 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 200112, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 80064 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.936 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2319, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.068207723, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000029412 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:12.937 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.938 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 0, 2024-04-22T06:48:12.939 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 1, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-22T06:48:12.940 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:12.941 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.942 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:12.943 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-22T06:48:12.944 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 13, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-22T06:48:12.945 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4197795, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 4197627, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 13 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 3, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 3, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 0, 2024-04-22T06:48:12.946 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 1, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 1, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 49528, 2024-04-22T06:48:12.947 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-22T06:48:12.948 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.949 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.951 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-22T06:48:12.952 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.953 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.954 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.955 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:12.956 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.957 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.958 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 5, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 3, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 4, 2024-04-22T06:48:12.960 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 3020, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:12.961 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:12.962 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:12.963 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 197297, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 52, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 9992, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 252, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:12.964 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 8512, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 13, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 1407, 2024-04-22T06:48:12.965 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 0, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 1407, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1407, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.765974596, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000544402 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1407, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1408 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.966 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 1407, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 1407, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 934, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 471, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-22T06:48:12.967 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-22T06:48:12.968 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-22T06:48:12.969 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 113, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 9, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 6, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 6, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-22T06:48:12.970 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:12.971 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 1548, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 453586, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 1548, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:12.972 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1548, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 453586, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:12.973 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 1407, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 3496, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1407, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3496, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:12.974 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 1407, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 1407, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1407, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1407, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.975 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:12.976 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x5640829a0780": { 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:12.977 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:12.978 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-22T06:48:12.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:12 smithi112 ceph-mon[31236]: pgmap v341: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-22T06:48:13.634 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-22T06:48:13.634 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-22T06:48:13.634 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-22T06:48:13.634 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:13.634 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:13.634 DEBUG:tasks.check_counter:Getting stats from f 2024-04-22T06:48:13.635 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-22T06:48:13.891 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:48:14.873 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:14 smithi177 ceph-mon[31346]: pgmap v342: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:14.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:14 smithi105 ceph-mon[27916]: pgmap v342: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:14.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:14 smithi112 ceph-mon[31236]: pgmap v342: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:15.500 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1606, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1606, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1382609, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 509894, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.159387386, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.050918333, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.042133476, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.050696206, 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:15.501 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1605, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.068987741, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000042983 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 448 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:15.502 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 291, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 243, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 1814280, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 135098, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.050321719, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.014888061, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.041492801, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005411552, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 242, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.016519304, 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000068261 2024-04-22T06:48:15.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 1812624, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 133760 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 732, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 890, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 179971, 2024-04-22T06:48:15.504 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 675757, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 4, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.081513500, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.033041418, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.016239218, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.023527365, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 887, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.043298730, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000048814 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.505 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 443616 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 3841, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.109267281, 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000028447 2024-04-22T06:48:15.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-22T06:48:15.507 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 31, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 5, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 26, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:15.508 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 24, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 2, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-22T06:48:15.509 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 4, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 234, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 6, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:15.510 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:15.511 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-22T06:48:15.512 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 226, 2024-04-22T06:48:15.513 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 13, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4860414, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 5099814, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 5099702, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.514 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 226 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 20, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 24, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 4, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 11, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 11, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 31, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 31, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-22T06:48:15.515 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 54040, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.516 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.517 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.518 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-22T06:48:15.519 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.521 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.522 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.523 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.524 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.525 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:15.526 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 5, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 3, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 4, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-22T06:48:15.527 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 2854, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:15.528 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:15.529 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 189780, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 62, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:15.530 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 9992, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 252, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 95998, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 774, 2024-04-22T06:48:15.531 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 2336, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 26, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 4, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 2336, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2336, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1.164961400, 2024-04-22T06:48:15.532 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000498699 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2336, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 2337 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 2336, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 2336, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 1478, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 855, 2024-04-22T06:48:15.533 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 2, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:15.534 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-22T06:48:15.535 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 113, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 24, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 5, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 5, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-22T06:48:15.536 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:15.537 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 2625, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 3123091, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:15.538 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 2625, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2625, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3123091, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:15.539 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2336, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 905520, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2336, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 905520, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:15.540 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 2336, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2336, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 2336, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2336, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.541 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.542 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x55afcaf426e0": { 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:15.543 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:15.544 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-22T06:48:16.385 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-22T06:48:16.385 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-22T06:48:16.385 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-22T06:48:16.385 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:16.385 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:16.385 DEBUG:tasks.check_counter:Getting stats from i 2024-04-22T06:48:16.385 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-22T06:48:16.629 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:48:16.867 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:16 smithi177 ceph-mon[31346]: pgmap v343: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:16.868 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:16 smithi177 ceph-mon[31346]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:48:16.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:16 smithi105 ceph-mon[27916]: pgmap v343: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:16.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:16 smithi105 ceph-mon[27916]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:48:16.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:16 smithi112 ceph-mon[31236]: pgmap v343: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:16.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:16 smithi112 ceph-mon[31236]: osdmap e114: 12 total, 12 up, 12 in 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 1419, 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 1421, 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 402141, 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 451249, 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 6, 2024-04-22T06:48:18.215 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.118088236, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.038002599, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.029387131, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.041558417, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1419, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.051230782, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000036103 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.216 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 0 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 7, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 103, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 3708, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 259338, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 4, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.007111841, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.004356566, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.001609635, 2024-04-22T06:48:18.217 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.000061300, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 102, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.007566262, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000074179 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 258640 2024-04-22T06:48:18.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 143, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 142, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 201116, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 81198, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 3, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 2, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.022066479, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.008730150, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.008701814, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.003434940, 2024-04-22T06:48:18.219 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 141, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.010084052, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000071518 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 200848, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 80880 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-22T06:48:18.220 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 2365, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.061749860, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000026109 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.221 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 0, 2024-04-22T06:48:18.222 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 1, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-22T06:48:18.223 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 0, 2024-04-22T06:48:18.224 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:18.225 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:18.226 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-22T06:48:18.227 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 10, 2024-04-22T06:48:18.228 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4196716, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 4196600, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.229 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 10 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 3, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 3, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 1, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 1, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-22T06:48:18.230 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 48708, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 215416 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.232 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.233 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.234 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.235 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.236 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-22T06:48:18.237 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.238 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.239 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.240 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.241 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.242 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 1, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 3, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 2, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 3, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 4303, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:18.243 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:18.244 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:18.245 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 181159, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 44, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-22T06:48:18.246 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 249, 2024-04-22T06:48:18.247 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 8512, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 13, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 1422, 2024-04-22T06:48:18.248 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 21, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 0, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 1422, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1422, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.749975135, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000527408 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1422, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 1423 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 1422, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 1422, 2024-04-22T06:48:18.249 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 944, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 476, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:18.250 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 1, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-22T06:48:18.251 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 114, 2024-04-22T06:48:18.252 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 10, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 4, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 4, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 2, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.253 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:18.254 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 1565, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 455839, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 1565, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1565, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 455839, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:18.255 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 1422, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 2417, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1422, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 2417, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 1422, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 1422, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1422, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 1422, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.257 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:18.258 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:18.259 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.260 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x56227d012780": { 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:18.261 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:18.262 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:18.263 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.264 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:18.264 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-22T06:48:18.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:18 smithi105 ceph-mon[27916]: pgmap v345: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:48:18.936 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:18 smithi177 ceph-mon[31346]: pgmap v345: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:48:18.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:18 smithi112 ceph-mon[31236]: pgmap v345: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-22T06:48:18.963 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-22T06:48:18.963 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-22T06:48:18.963 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-04-22T06:48:18.963 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:18.963 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:18.963 DEBUG:tasks.check_counter:Getting stats from l 2024-04-22T06:48:18.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697 shell --fsid 48924e24-0071-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-22T06:48:19.199 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/mon.c/config 2024-04-22T06:48:20.797 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:20 smithi177 ceph-mon[31346]: pgmap v346: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-0": { 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 128, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 160, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 35227, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 45735, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 13, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 3, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.018291013, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.005051515, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.007913680, 2024-04-22T06:48:20.798 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.003259684, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 156, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.009298643, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000059606 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 54, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000012306, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000227 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 944, 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 4912 2024-04-22T06:48:20.799 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-1": { 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 202, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 188, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 295337, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 117299, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 7, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 5, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.028675453, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.010735484, 2024-04-22T06:48:20.800 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.012406472, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.004417782, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 189, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.016654405, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000088118 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 16, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000002430, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000151 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 265056, 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 93488 2024-04-22T06:48:20.801 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "AsyncMessenger::Worker-2": { 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_messages": 240, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages": 465, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_bytes": 100774, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_bytes": 518701, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_created_connections": 8, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_active_connections": 6, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_total_time": 0.044352198, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_send_time": 0.018985946, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_recv_time": 0.016082688, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_running_fast_dispatch_time": 0.005689998, 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_messages_queue_lat": { 2024-04-22T06:48:20.802 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 464, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.031827021, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000068592 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_handle_ack_lat": { 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 444, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000175579, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000395 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "msgr_send_encrypted_bytes": 278352 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "cct": { 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "total_workers": 1, 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: "unhealthy_workers": 0 2024-04-22T06:48:20.803 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-MDSRank": { 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 304, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.012590340, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000041415 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "finisher-PurgeQueue": { 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "queue_len": 0, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "complete_latency": { 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 4, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000107687, 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000026921 2024-04-22T06:48:20.804 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "mds": { 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "request": 0, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "reply": 0, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "reply_latency": { 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_reply": 0, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "forward": 0, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_complete": 11, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_fetch_keys": 10, 2024-04-22T06:48:20.805 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_commit": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_split": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_merge": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes": 10, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_top": 7, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_bottom": 3, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pin_tail": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_pinned": 10, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_expired": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "inodes_with_caps": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "caps": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "subtrees": 1, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_hit": 0, 2024-04-22T06:48:20.806 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_forward": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_discover": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_dir_fetch": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_remote_ino": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "traverse_lock": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "load_cent": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "q": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "exported": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "exported_inodes": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "imported": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "imported_inodes": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_dir_fetch": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_backtrace_fetch": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "openino_peer_discover": 0, 2024-04-22T06:48:20.807 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rfiles": 4, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rbytes": 234, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "root_rsnaps": 3, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_fetch": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_set_tag": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_backtrace_repaired": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_inotable_repaired": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_inodes": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dir_base_inodes": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_dirfrag_rstats": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "scrub_file_inodes": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_inode_file_caps": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_revoke": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_grant": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_trunc": 0, 2024-04-22T06:48:20.808 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_caps_dirty": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_cap_release": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "process_request_cap_release": 0 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_cache": { 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_update_receipt": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_try_discover": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_send_discover": 1, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "dir_handle_discover": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_delayed": 0, 2024-04-22T06:48:20.809 INFO:teuthology.orchestra.run.smithi177.stdout: "num_strays_enqueuing": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_created": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_enqueued": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_reintegrated": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "strays_migrated": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_processing": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_enqueued": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "num_recovering_prioritized": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_started": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "recovery_completed": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_path": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_quiesce_inode": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_enqueue_scrub": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_exportdir": 0, 2024-04-22T06:48:20.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_flush": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragmentdir": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_fragstats": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "ireq_inodestats": 0 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_log": { 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evlrg": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evadd": 1, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evex": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evtrm": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "ev": 13, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evexg": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "evexd": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "segadd": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "segex": 0, 2024-04-22T06:48:20.811 INFO:teuthology.orchestra.run.smithi177.stdout: "segtrm": 0, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "seg": 2, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "segmjr": 2, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "segexg": 0, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "segexd": 0, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "expos": 4194304, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "wrpos": 4197795, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "rdpos": 4197459, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "jlat": { 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 1, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.002329542, 2024-04-22T06:48:20.812 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002329542 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "replayed": 12 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_mem": { 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "ino": 13, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "ino+": 14, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "ino-": 1, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dir": 11, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dir+": 12, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dir-": 1, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dn": 10, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dn+": 10, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "dn-": 0, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "cap": 0, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "cap+": 0, 2024-04-22T06:48:20.813 INFO:teuthology.orchestra.run.smithi177.stdout: "cap-": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "rss": 55684, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "heap": 239992 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_server": { 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_client_request": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "dispatch_server_request": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_request": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_client_session": 11, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "handle_peer_request": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "req_create_latency": { 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.814 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getattr_latency": { 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getfilelock_latency": { 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "req_link_latency": { 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.815 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookup_latency": { 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookuphash_latency": { 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupino_latency": { 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.816 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupname_latency": { 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupparent_latency": { 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lookupsnap_latency": { 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.817 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "req_lssnap_latency": { 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mkdir_latency": { 2024-04-22T06:48:20.818 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mknod_latency": { 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "req_mksnap_latency": { 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.819 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "req_open_latency": { 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "req_readdir_latency": { 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rename_latency": { 2024-04-22T06:48:20.820 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "req_renamesnap_latency": { 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "req_snapdiff_latency": { 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmdir_latency": { 2024-04-22T06:48:20.821 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmsnap_latency": { 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "req_rmxattr_latency": { 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.822 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setattr_latency": { 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setdirlayout_latency": { 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setfilelock_latency": { 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.823 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setlayout_latency": { 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "req_setxattr_latency": { 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "req_symlink_latency": { 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.824 INFO:teuthology.orchestra.run.smithi177.stdout: "req_unlink_latency": { 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_revoke_eviction": 0, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "cap_acquisition_throttle": 0, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "req_getvxattr_latency": { 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_sessions": { 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "session_count": 2, 2024-04-22T06:48:20.825 INFO:teuthology.orchestra.run.smithi177.stdout: "session_add": 5, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "session_remove": 3, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_open": 4, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "sessions_stale": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "total_load": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "average_load": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "avg_session_uptime": 2970, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "mdthresh_evicted": 0 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "mempool": { 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_bytes": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "bloom_filter_items": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_bytes": 0, 2024-04-22T06:48:20.826 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_alloc_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_data_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_onode_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_meta_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_other_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_cache_buffer_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_extent_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_bytes": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_blob_items": 0, 2024-04-22T06:48:20.827 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_shared_blob_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_inline_bl_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_fsck_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_txc_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_deferred_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluestore_writing_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_bytes": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_reader_items": 0, 2024-04-22T06:48:20.828 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs_file_writer_items": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_bytes": 262487, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_anon_items": 90, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "buffer_meta_items": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_items": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_mapbl_items": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_pglog_items": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_bytes": 9872, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_items": 249, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_bytes": 0, 2024-04-22T06:48:20.829 INFO:teuthology.orchestra.run.smithi177.stdout: "osdmap_mapping_items": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_bytes": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "pgmap_items": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_bytes": 52984, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "mds_co_items": 175, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_bytes": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_1_items": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_bytes": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "unittest_2_items": 0 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "objecter": { 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "op_active": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "op_laggy": 0, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send": 198, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "op_send_bytes": 679, 2024-04-22T06:48:20.830 INFO:teuthology.orchestra.run.smithi177.stdout: "op_resend": 0, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_reply": 198, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_latency": { 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 198, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.432985696, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.002186796 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_inflight": 0, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "oplen_avg": { 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 198, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 242 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op": 198, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_r": 189, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_w": 9, 2024-04-22T06:48:20.831 INFO:teuthology.orchestra.run.smithi177.stdout: "op_rmw": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "op_pg": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_stat": 107, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_create": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_read": 58, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_write": 1, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writefull": 1, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_writesame": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_append": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_zero": 1, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_truncate": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_delete": 5, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_mapext": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_sparse_read": 0, 2024-04-22T06:48:20.832 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_clonerange": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_getxattr": 22, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_setxattr": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_cmpxattr": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_rmxattr": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_resetxattrs": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_call": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_watch": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_notify": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_src_cmpxattr": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_pgls_filter": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "osdop_other": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_active": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_send": 0, 2024-04-22T06:48:20.833 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_resend": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "linger_ping": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_active": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_send": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolop_resend": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_active": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_send": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "poolstat_resend": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_active": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_send": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "statfs_resend": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "command_active": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "command_send": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "command_resend": 0, 2024-04-22T06:48:20.834 INFO:teuthology.orchestra.run.smithi177.stdout: "map_epoch": 114, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "map_full": 0, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "map_inc": 14, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_sessions": 9, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_open": 9, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_session_close": 0, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_laggy": 0, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_wr": 1, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_rd": 46, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_del": 0 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "oft": { 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_objs": 0, 2024-04-22T06:48:20.835 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_kv_pairs": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_updates": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "omap_total_removes": 0 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "osd-slow-ops": { 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "slow_ops_count": 0 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "purge_queue": { 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_ops_high_water": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executing_high_water": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed_ops": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_executed": 0, 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: "pq_item_in_journal": 0 2024-04-22T06:48:20.836 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 566, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 368120, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 566, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 566, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 368120, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.837 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_bytes": { 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 104857600, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 198, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 3754, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 198, 2024-04-22T06:48:20.838 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 3754, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-objecter_ops": { 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 1024, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:20.839 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 198, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 198, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 198, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 198, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle": { 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:20.840 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-04-22T06:48:20.841 INFO:teuthology.orchestra.run.smithi177.stdout: "throttle-write_buf_throttle-0x561b80740780": { 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "val": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "max": 3758096384, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "get_started": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "get": 1, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "get_sum": 168, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_fail": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "get_or_fail_success": 1, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "take": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "take_sum": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "put": 1, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "put_sum": 168, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "wait": { 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "avgcount": 0, 2024-04-22T06:48:20.842 INFO:teuthology.orchestra.run.smithi177.stdout: "sum": 0.000000000, 2024-04-22T06:48:20.843 INFO:teuthology.orchestra.run.smithi177.stdout: "avgtime": 0.000000000 2024-04-22T06:48:20.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.843 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-04-22T06:48:20.843 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-04-22T06:48:20.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:20 smithi105 ceph-mon[27916]: pgmap v346: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2024-04-22T06:48:20.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:20 smithi112 ceph-mon[31236]: pgmap v346: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2024-04-22T06:48:21.596 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-22T06:48:21.596 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-22T06:48:21.596 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=3 2024-04-22T06:48:21.596 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-22T06:48:21.596 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-22T06:48:21.597 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-22T06:48:21.608 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:48:21.609 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:48:21.609 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:48:21.626 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:48:21.626 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:48:21.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.827+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/2612075397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 msgr2=0x7fcc28069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.827+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/2612075397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28069560 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009900 tx=0x7fcc1802bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.827+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/2612075397 shutdown_connections 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.827+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/2612075397 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc28069aa0 0x7fcc2810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/2612075397 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28069560 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/2612075397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc28107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/2612075397 >> 172.21.15.105:0/2612075397 conn(0x7fcc28077e30 msgr2=0x7fcc28078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:21.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/2612075397 shutdown_connections 2024-04-22T06:48:21.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/2612075397 wait complete. 2024-04-22T06:48:21.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 Processor -- start 2024-04-22T06:48:21.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 -- start start 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.828+0000 7fcc2e8d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc28069aa0 0x7fcc28198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc2819cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcc281121f0 con 0x7fcc280690e0 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28112070 con 0x7fcc28106f50 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28112370 con 0x7fcc28069aa0 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37766/0 (socket says 172.21.15.105:37766) 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 -- 172.21.15.105:0/841493856 learned_addr learned my addr 172.21.15.105:0/841493856 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:21.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2ce76640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc2819cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 -- 172.21.15.105:0/841493856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc28069aa0 msgr2=0x7fcc28198c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc28069aa0 0x7fcc28198c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 -- 172.21.15.105:0/841493856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 msgr2=0x7fcc2819cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc2819cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 -- 172.21.15.105:0/841493856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2819d700 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2ce76640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc2819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc27fff640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00d950 tx=0x7fcc1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c017070 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2819d9f0 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc1c009c30 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c00f620 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.829+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc281a5300 con 0x7fcc280690e0 2024-04-22T06:48:21.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.830+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc28068de0 con 0x7fcc280690e0 2024-04-22T06:48:21.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.831+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fcc1c010460 con 0x7fcc280690e0 2024-04-22T06:48:21.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.832+0000 7fcc257fa640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 0x7fcc100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:21.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.832+0000 7fcc277fe640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 0x7fcc100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:21.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.832+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fcc1c0d08d0 con 0x7fcc280690e0 2024-04-22T06:48:21.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.833+0000 7fcc277fe640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 0x7fcc100a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcc18036040 tx=0x7fcc18005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:21.835 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:21.837+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc1c09cbb0 con 0x7fcc280690e0 2024-04-22T06:48:22.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.020+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc28109f10 con 0x7fcc280690e0 2024-04-22T06:48:22.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.021+0000 7fcc257fa640 1 -- 172.21.15.105:0/841493856 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+24654 (secure 0 0 0) 0x7fcc1c009de0 con 0x7fcc280690e0 2024-04-22T06:48:22.020 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:48:22.020 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":114,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:48:16.161139+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":113,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6837/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6839/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.177:6838/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.112:6836/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.177:6837/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000","172.21.15.177:6832/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6833/583171810":"2024-04-23T06:39:27.429757+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.112:6836/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.112:6837/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.112:6837/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.105:6840/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6841/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6841/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6838/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6838/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.177:6839/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.105:6840/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6839/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6836/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6833/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6834/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.112:6835/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.177:6836/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.112:6832/583171810":"2024-04-23T06:39:27.429757+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 msgr2=0x7fcc100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 0x7fcc100a61e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcc18036040 tx=0x7fcc18005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 msgr2=0x7fcc28198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c00d950 tx=0x7fcc1c00de10 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 shutdown_connections 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcc100a3d20 0x7fcc100a61e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc28106f50 0x7fcc2819cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcc28069aa0 0x7fcc28198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 --2- 172.21.15.105:0/841493856 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcc280690e0 0x7fcc28198710 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 >> 172.21.15.105:0/841493856 conn(0x7fcc28077e30 msgr2=0x7fcc280fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 shutdown_connections 2024-04-22T06:48:22.022 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.023+0000 7fcc2e8d9640 1 -- 172.21.15.105:0/841493856 wait complete. 2024-04-22T06:48:22.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T06:31:10.628361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-22T06:35:38.535076+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-22T06:35:39.107386+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 113, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-22T06:48:22.033 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2396194692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4107250 msgr2=0x7f4fd410ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2396194692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4107250 0x7f4fd410ee20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009ca0 tx=0x7f4fcc02c120 comp rx=0 tx=0).stop 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2396194692 shutdown_connections 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2396194692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4107250 0x7f4fd410ee20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2396194692 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4102b30 0x7f4fd4106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2396194692 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd4102180 0x7f4fd4102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2396194692 >> 172.21.15.105:0/2396194692 conn(0x7f4fd40fd9c0 msgr2=0x7f4fd40ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2396194692 shutdown_connections 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2396194692 wait complete. 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 Processor -- start 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 -- start start 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.138+0000 7f4fdbdb6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4107250 0x7f4fd419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd4198760 0x7f4fd4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41122b0 con 0x7f4fd4102180 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd4112130 con 0x7f4fd4107250 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd4112430 con 0x7f4fd4198760 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.139 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37784/0 (socket says 172.21.15.105:37784) 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 -- 172.21.15.105:0/2054301282 learned_addr learned my addr 172.21.15.105:0/2054301282 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9351640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4107250 0x7f4fd419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 -- 172.21.15.105:0/2054301282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd4198760 msgr2=0x7f4fd4198be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd4198760 0x7f4fd4198be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 -- 172.21.15.105:0/2054301282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4107250 msgr2=0x7f4fd419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4107250 0x7f4fd419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 -- 172.21.15.105:0/2054301282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd4199360 con 0x7f4fd4102180 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fd9b52640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000b450 tx=0x7f4fd000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0015020 con 0x7f4fd4102180 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd41a53f0 con 0x7f4fd4102180 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd41a57f0 con 0x7f4fd4102180 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0011d40 con 0x7f4fd4102180 2024-04-22T06:48:22.140 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.139+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd001adf0 con 0x7f4fd4102180 2024-04-22T06:48:22.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.140+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb8005290 con 0x7f4fd4102180 2024-04-22T06:48:22.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.141+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f4fd0018020 con 0x7f4fd4102180 2024-04-22T06:48:22.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.142+0000 7f4fcaffd640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 0x7f4fc00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.142+0000 7f4fd9351640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 0x7f4fc00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.142+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7f4fd00d04a0 con 0x7f4fd4102180 2024-04-22T06:48:22.142 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.142+0000 7f4fd9351640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 0x7f4fc00a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4199f60 tx=0x7f4fc4008040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.144 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.146+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fd001f020 con 0x7f4fd4102180 2024-04-22T06:48:22.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.323+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4fb8005850 con 0x7f4fd4102180 2024-04-22T06:48:22.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.323+0000 7f4fcaffd640 1 -- 172.21.15.105:0/2054301282 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) ==== 93+0+10 (secure 0 0 0) 0x7f4fd000d040 con 0x7f4fd4102180 2024-04-22T06:48:22.321 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 msgr2=0x7f4fc00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 0x7f4fc00a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4199f60 tx=0x7f4fc4008040 comp rx=0 tx=0).stop 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 msgr2=0x7f4fd419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000b450 tx=0x7f4fd000b910 comp rx=0 tx=0).stop 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 shutdown_connections 2024-04-22T06:48:22.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f4fc00a3d20 0x7f4fc00a61e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4fd4198760 0x7f4fd4198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4fd4107250 0x7f4fd419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 --2- 172.21.15.105:0/2054301282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4fd4102180 0x7f4fd419e650 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 >> 172.21.15.105:0/2054301282 conn(0x7f4fd40fd9c0 msgr2=0x7f4fd40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 shutdown_connections 2024-04-22T06:48:22.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.325+0000 7f4fdbdb6640 1 -- 172.21.15.105:0/2054301282 wait complete. 2024-04-22T06:48:22.334 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-22T06:48:22.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 -- 172.21.15.105:0/2432956655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 msgr2=0x7f240810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 --2- 172.21.15.105:0/2432956655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f240810dd40 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009ca0 tx=0x7f23f802c120 comp rx=0 tx=0).stop 2024-04-22T06:48:22.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 -- 172.21.15.105:0/2432956655 shutdown_connections 2024-04-22T06:48:22.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 --2- 172.21.15.105:0/2432956655 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f240810dd40 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 --2- 172.21.15.105:0/2432956655 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2408101a50 0x7f2408105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 --2- 172.21.15.105:0/2432956655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f24081010a0 0x7f2408101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 -- 172.21.15.105:0/2432956655 >> 172.21.15.105:0/2432956655 conn(0x7f24080771d0 msgr2=0x7f24080775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 -- 172.21.15.105:0/2432956655 shutdown_connections 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.437+0000 7f240cb7e640 1 -- 172.21.15.105:0/2432956655 wait complete. 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 Processor -- start 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 -- start start 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24081010a0 0x7f2408198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 0x7f240819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.438 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2408112230 con 0x7f2408106170 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f24081120b0 con 0x7f240819d010 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f240cb7e640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f24081123b0 con 0x7f24081010a0 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37810/0 (socket says 172.21.15.105:37810) 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 -- 172.21.15.105:0/1833404432 learned_addr learned my addr 172.21.15.105:0/1833404432 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2406d9d640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 0x7f240819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 -- 172.21.15.105:0/1833404432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24081010a0 msgr2=0x7f2408198750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24081010a0 0x7f2408198750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 -- 172.21.15.105:0/1833404432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 msgr2=0x7f240819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 0x7f240819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2405d9b640 1 -- 172.21.15.105:0/1833404432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240819db00 con 0x7f2408106170 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.438+0000 7f2406d9d640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 0x7f240819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f2405d9b640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00b470 tx=0x7f23fc00b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.439 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc015020 con 0x7f2408106170 2024-04-22T06:48:22.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23fc011d40 con 0x7f2408106170 2024-04-22T06:48:22.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240819ddf0 con 0x7f2408106170 2024-04-22T06:48:22.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00f340 con 0x7f2408106170 2024-04-22T06:48:22.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.439+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24081a5590 con 0x7f2408106170 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.440+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24080691c0 con 0x7f2408106170 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.441+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f23fc018020 con 0x7f2408106170 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.441+0000 7f23f77fe640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 0x7f23f00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.441+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7f23fc0d0bc0 con 0x7f2408106170 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.441+0000 7f240659c640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 0x7f23f00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.442+0000 7f240659c640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 0x7f23f00a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f240000b990 tx=0x7f240000b370 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.445+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23fc01f020 con 0x7f2408106170 2024-04-22T06:48:22.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.622+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2408101a50 con 0x7f2408106170 2024-04-22T06:48:22.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.622+0000 7f23f77fe640 1 -- 172.21.15.105:0/1833404432 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) ==== 104+0+11 (secure 0 0 0) 0x7f23fc00d040 con 0x7f2408106170 2024-04-22T06:48:22.621 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 32 2024-04-22T06:48:22.623 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 msgr2=0x7f23f00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.623 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 0x7f23f00a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f240000b990 tx=0x7f240000b370 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 msgr2=0x7f2408198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00b470 tx=0x7f23fc00b930 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 shutdown_connections 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f23f00a3d20 0x7f23f00a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f240819d010 0x7f240819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2408106170 0x7f2408198c90 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 --2- 172.21.15.105:0/1833404432 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f24081010a0 0x7f2408198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.624 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 >> 172.21.15.105:0/1833404432 conn(0x7f24080771d0 msgr2=0x7f24080feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.625 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 shutdown_connections 2024-04-22T06:48:22.625 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.625+0000 7f240cb7e640 1 -- 172.21.15.105:0/1833404432 wait complete. 2024-04-22T06:48:22.635 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/827730481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58102190 msgr2=0x7fdd58102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/827730481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58102190 0x7fdd58102570 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009900 tx=0x7fdd5002bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/827730481 shutdown_connections 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/827730481 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd58107260 0x7fdd5810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/827730481 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58102b40 0x7fdd58106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/827730481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58102190 0x7fdd58102570 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.737 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/827730481 >> 172.21.15.105:0/827730481 conn(0x7fdd580fd990 msgr2=0x7fdd580ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/827730481 shutdown_connections 2024-04-22T06:48:22.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/827730481 wait complete. 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.739+0000 7fdd5d1a4640 1 Processor -- start 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 -- start start 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd58102190 0x7fdd5819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58107260 0x7fdd5819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd58112240 con 0x7fdd58198660 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd581120c0 con 0x7fdd58107260 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5d1a4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd581123c0 con 0x7fdd58102190 2024-04-22T06:48:22.740 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37816/0 (socket says 172.21.15.105:37816) 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 -- 172.21.15.105:0/447063842 learned_addr learned my addr 172.21.15.105:0/447063842 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5659c640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58107260 0x7fdd5819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 -- 172.21.15.105:0/447063842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd58102190 msgr2=0x7fdd5819e5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd58102190 0x7fdd5819e5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 -- 172.21.15.105:0/447063842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58107260 msgr2=0x7fdd5819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58107260 0x7fdd5819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 -- 172.21.15.105:0/447063842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd58199260 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd5759e640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b3f0 tx=0x7fdd4800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.740+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48015020 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.741+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd581a5340 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.741+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd48011ce0 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.741+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd581a57a0 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.741+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd48010da0 con 0x7fdd58198660 2024-04-22T06:48:22.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.741+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd58103660 con 0x7fdd58198660 2024-04-22T06:48:22.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.743+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fdd48018020 con 0x7fdd58198660 2024-04-22T06:48:22.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.743+0000 7fdd3ffff640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 0x7fdd400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:22.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.743+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fdd480d08a0 con 0x7fdd58198660 2024-04-22T06:48:22.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.743+0000 7fdd56d9d640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 0x7fdd400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:22.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.744+0000 7fdd56d9d640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 0x7fdd400a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd50036040 tx=0x7fdd50005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:22.745 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.747+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd4801f020 con 0x7fdd58198660 2024-04-22T06:48:22.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:22 smithi105 ceph-mon[27916]: pgmap v347: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:22 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/841493856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:22 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2054301282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:48:22.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:22 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1833404432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:48:22.925 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.927+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd5810a490 con 0x7fdd58198660 2024-04-22T06:48:22.926 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.928+0000 7fdd3ffff640 1 -- 172.21.15.105:0/447063842 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) ==== 100+0+11 (secure 0 0 0) 0x7fdd4809cc90 con 0x7fdd58198660 2024-04-22T06:48:22.926 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 64 2024-04-22T06:48:22.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 msgr2=0x7fdd400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 0x7fdd400a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd50036040 tx=0x7fdd50005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 msgr2=0x7fdd58198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:22.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b3f0 tx=0x7fdd4800b8b0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 shutdown_connections 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd400a3d20 0x7fdd400a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd58198660 0x7fdd58198ae0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd58107260 0x7fdd5819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 --2- 172.21.15.105:0/447063842 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd58102190 0x7fdd5819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 >> 172.21.15.105:0/447063842 conn(0x7fdd580fd990 msgr2=0x7fdd580fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 shutdown_connections 2024-04-22T06:48:22.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:22.930+0000 7fdd5d1a4640 1 -- 172.21.15.105:0/447063842 wait complete. 2024-04-22T06:48:22.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-22T06:48:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:22 smithi112 ceph-mon[31236]: pgmap v347: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:22 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/841493856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:22 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2054301282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:48:22.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:22 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1833404432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:48:22.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:22 smithi177 ceph-mon[31346]: pgmap v347: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-22T06:48:22.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:22 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/841493856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:22.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:22 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2054301282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T06:48:22.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:22 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1833404432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-22T06:48:23.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- 172.21.15.105:0/3735551216 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd80101a50 msgr2=0x7fdd80105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 --2- 172.21.15.105:0/3735551216 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd80101a50 0x7fdd80105a90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009900 tx=0x7fdd6c02bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:23.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- 172.21.15.105:0/3735551216 shutdown_connections 2024-04-22T06:48:23.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 --2- 172.21.15.105:0/3735551216 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd80106170 0x7fdd8010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 --2- 172.21.15.105:0/3735551216 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd80101a50 0x7fdd80105a90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 --2- 172.21.15.105:0/3735551216 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd801010a0 0x7fdd80101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- 172.21.15.105:0/3735551216 >> 172.21.15.105:0/3735551216 conn(0x7fdd80077eb0 msgr2=0x7fdd800782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- 172.21.15.105:0/3735551216 shutdown_connections 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- 172.21.15.105:0/3735551216 wait complete. 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 Processor -- start 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.042+0000 7fdd87206640 1 -- start start 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd80106170 0x7fdd80197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd8019bf70 0x7fdd8019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd80111130 con 0x7fdd801010a0 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdd80110fb0 con 0x7fdd8019bf70 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801112b0 con 0x7fdd80106170 2024-04-22T06:48:23.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37844/0 (socket says 172.21.15.105:37844) 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 -- 172.21.15.105:0/1734411580 learned_addr learned my addr 172.21.15.105:0/1734411580 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd857a3640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd8019bf70 0x7fdd8019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 -- 172.21.15.105:0/1734411580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd80106170 msgr2=0x7fdd80197bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd80106170 0x7fdd80197bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 -- 172.21.15.105:0/1734411580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd8019bf70 msgr2=0x7fdd8019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd8019bf70 0x7fdd8019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 -- 172.21.15.105:0/1734411580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8019ca60 con 0x7fdd801010a0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd84fa2640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400b450 tx=0x7fdd7400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74015020 con 0x7fdd801010a0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8019cd50 con 0x7fdd801010a0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd74011d40 con 0x7fdd801010a0 2024-04-22T06:48:23.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.043+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400f340 con 0x7fdd801010a0 2024-04-22T06:48:23.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.044+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd801a44f0 con 0x7fdd801010a0 2024-04-22T06:48:23.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.045+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd800691c0 con 0x7fdd801010a0 2024-04-22T06:48:23.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.046+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fdd74018020 con 0x7fdd801010a0 2024-04-22T06:48:23.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.046+0000 7fdd7dffb640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.046+0000 7fdd7ffff640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.047+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fdd740d0640 con 0x7fdd801010a0 2024-04-22T06:48:23.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.047+0000 7fdd7ffff640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 0x7fdd700a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c036040 tx=0x7fdd6c005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:23.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.051+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd7401f020 con 0x7fdd801010a0 2024-04-22T06:48:23.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.239+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdd80109460 con 0x7fdd801010a0 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.239+0000 7fdd7dffb640 1 -- 172.21.15.105:0/1734411580 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 40 entries v114) ==== 81+0+2435 (secure 0 0 0) 0x7fdd7400d040 con 0x7fdd801010a0 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6837/678281724 2024-04-23T06:40:22.020245+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6839/3064875873 2024-04-23T06:40:22.014751+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6838/3064875873 2024-04-23T06:40:22.014751+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/3300341732 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/1014082495 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6836/1826997924 2024-04-23T06:40:20.987927+0000 2024-04-22T06:48:23.238 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6801/2556874834 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6800/2556874834 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6837/4114728122 2024-04-23T06:38:36.724579+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6801/3871303088 2024-04-23T06:28:44.539275+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/817153020 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/937171571 2024-04-23T06:28:44.539275+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/768203659 2024-04-23T06:28:17.391954+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6832/624977252 2024-04-23T06:39:26.466860+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6833/583171810 2024-04-23T06:39:27.429757+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/3406296347 2024-04-23T06:28:44.539275+0000 2024-04-22T06:48:23.239 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6800/3871303088 2024-04-23T06:28:44.539275+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/2711052846 2024-04-23T06:28:17.391954+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/1361172178 2024-04-23T06:28:17.391954+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6836/3970644798 2024-04-23T06:38:36.718748+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6837/1826997924 2024-04-23T06:40:20.987927+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6837/3970644798 2024-04-23T06:38:36.718748+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6840/2147123680 2024-04-23T06:38:36.825169+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6801/2332170576 2024-04-23T06:28:17.391954+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/3570614146 2024-04-23T06:37:36.925007+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6841/85174468 2024-04-23T06:40:21.258556+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6841/2147123680 2024-04-23T06:38:36.825169+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6838/256609249 2024-04-23T06:39:26.670520+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6838/1381266768 2024-04-23T06:38:37.426711+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6839/1381266768 2024-04-23T06:38:37.426711+0000 2024-04-22T06:48:23.240 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6840/85174468 2024-04-23T06:40:21.258556+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6839/256609249 2024-04-23T06:39:26.670520+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6836/678281724 2024-04-23T06:40:22.020245+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6833/624977252 2024-04-23T06:39:26.466860+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6834/3883368202 2024-04-23T06:39:27.201039+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:6800/2332170576 2024-04-23T06:28:17.391954+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105:0/1118894802 2024-04-23T06:28:44.539275+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6835/3883368202 2024-04-23T06:39:27.201039+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.177:6836/4114728122 2024-04-23T06:38:36.724579+0000 2024-04-22T06:48:23.241 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.112:6832/583171810 2024-04-23T06:39:27.429757+0000 2024-04-22T06:48:23.242 INFO:teuthology.orchestra.run.smithi105.stderr:listed 40 entries 2024-04-22T06:48:23.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 msgr2=0x7fdd700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 0x7fdd700a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c036040 tx=0x7fdd6c005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 msgr2=0x7fdd801976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400b450 tx=0x7fdd7400b910 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 shutdown_connections 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fdd700a3d20 0x7fdd700a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdd8019bf70 0x7fdd8019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdd80106170 0x7fdd80197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.241+0000 7fdd87206640 1 --2- 172.21.15.105:0/1734411580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd801010a0 0x7fdd801976b0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.242+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 >> 172.21.15.105:0/1734411580 conn(0x7fdd80077eb0 msgr2=0x7fdd80100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.242+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 shutdown_connections 2024-04-22T06:48:23.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.242+0000 7fdd87206640 1 -- 172.21.15.105:0/1734411580 wait complete. 2024-04-22T06:48:23.250 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-22T06:48:23.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.251 DEBUG:teuthology.orchestra.run.smithi105:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-22T06:48:23.290 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:48:23.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.291 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-22T06:48:23.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.352 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-22T06:48:23.406 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-22T06:48:23.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-22T06:48:23.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link delete ceph-brx 2024-04-22T06:48:23.558 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-22T06:48:23.559 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-22T06:48:23.559 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-22T06:48:23.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056106,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=185 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/e3d4fe72944bc6806d2ddaaae1bd207e95fb0a750a029b88c0e9ff2f93f3f574/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/e3d4fe72944bc6806d2ddaaae1bd207e95fb0a750a029b88c0e9ff2f93f3f574/diff,workdir=/var/lib/containers/storage/overlay/e3d4fe72944bc6806d2ddaaae1bd207e95fb0a750a029b88c0e9ff2f93f3f574/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/90e74cc0324e92505f87ed653d2ad92b6940aa3189a446dcd2c7afb4bf52df21/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/90e74cc0324e92505f87ed653d2ad92b6940aa3189a446dcd2c7afb4bf52df21/diff,workdir=/var/lib/containers/storage/overlay/90e74cc0324e92505f87ed653d2ad92b6940aa3189a446dcd2c7afb4bf52df21/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/1420345a501e37e659e5b256065258da40f487c6c3bab81c33894e8c185a9325/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/1420345a501e37e659e5b256065258da40f487c6c3bab81c33894e8c185a9325/diff,workdir=/var/lib/containers/storage/overlay/1420345a501e37e659e5b256065258da40f487c6c3bab81c33894e8c185a9325/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c49b8dac48f50d0d3f34cc9dab36d71db15ff5691418d888c52ba0b4572ddbfd/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/c49b8dac48f50d0d3f34cc9dab36d71db15ff5691418d888c52ba0b4572ddbfd/diff,workdir=/var/lib/containers/storage/overlay/c49b8dac48f50d0d3f34cc9dab36d71db15ff5691418d888c52ba0b4572ddbfd/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7078112d2025222ab3de9dfd850d3dc79b344eb904f5a9ea2b891bbc8ef1b858/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/7078112d2025222ab3de9dfd850d3dc79b344eb904f5a9ea2b891bbc8ef1b858/diff,workdir=/var/lib/containers/storage/overlay/7078112d2025222ab3de9dfd850d3dc79b344eb904f5a9ea2b891bbc8ef1b858/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2d1ca434682d174f7442d743470657940a6a0370b2c7e73955ed74bdab6cd807/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/2d1ca434682d174f7442d743470657940a6a0370b2c7e73955ed74bdab6cd807/diff,workdir=/var/lib/containers/storage/overlay/2d1ca434682d174f7442d743470657940a6a0370b2c7e73955ed74bdab6cd807/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b93f4c5cf360e59ca93e76ce969248cc1e1b663e69b0cb31246e736a71fd7ba5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/b93f4c5cf360e59ca93e76ce969248cc1e1b663e69b0cb31246e736a71fd7ba5/diff,workdir=/var/lib/containers/storage/overlay/b93f4c5cf360e59ca93e76ce969248cc1e1b663e69b0cb31246e736a71fd7ba5/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/766f268fda2690b6cb4b78a02755678d05635509702359d0ce3a9f4057270190/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/766f268fda2690b6cb4b78a02755678d05635509702359d0ce3a9f4057270190/diff,workdir=/var/lib/containers/storage/overlay/766f268fda2690b6cb4b78a02755678d05635509702359d0ce3a9f4057270190/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 overlay /var/lib/containers/storage/overlay/b2c9a6abed0608c9bca6459a1c359835d5b495294df275268b88423642728ce2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/b2c9a6abed0608c9bca6459a1c359835d5b495294df275268b88423642728ce2/diff,workdir=/var/lib/containers/storage/overlay/b2c9a6abed0608c9bca6459a1c359835d5b495294df275268b88423642728ce2/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6a9c4c98725a3389c27cc29e9f5872489e061db6038ffcf790ab51b6f5397dbd/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/U3H5CNKDA7PYCMA6JSAMJXEKM3:/var/lib/containers/storage/overlay/l/CX4NRTPVPVVZLCXDXPHSYKUNL4,upperdir=/var/lib/containers/storage/overlay/6a9c4c98725a3389c27cc29e9f5872489e061db6038ffcf790ab51b6f5397dbd/diff,workdir=/var/lib/containers/storage/overlay/6a9c4c98725a3389c27cc29e9f5872489e061db6038ffcf790ab51b6f5397dbd/work,metacopy=on,volatile 0 0 2024-04-22T06:48:23.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.575 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-04-22T06:48:23.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-22T06:48:23.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link delete ceph-brx 2024-04-22T06:48:23.697 INFO:teuthology.orchestra.run.smithi105.stderr:Cannot find device "ceph-brx" 2024-04-22T06:48:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:48:23.699 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T06:48:23.710 INFO:tasks.cephadm:Teardown begin 2024-04-22T06:48:23.710 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-22T06:48:23.711 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:23.840 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/2728467019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d40690e0 msgr2=0x7fb6d4069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/2728467019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d40690e0 0x7fb6d4069560 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009930 tx=0x7fb6c402bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/2728467019 shutdown_connections 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/2728467019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4069aa0 0x7fb6d410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/2728467019 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d40690e0 0x7fb6d4069560 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/2728467019 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6d4107fd0 0x7fb6d41083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/2728467019 >> 172.21.15.105:0/2728467019 conn(0x7fb6d4077e30 msgr2=0x7fb6d4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/2728467019 shutdown_connections 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/2728467019 wait complete. 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 Processor -- start 2024-04-22T06:48:23.841 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.840+0000 7fb6dbdd5640 1 -- start start 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6d40690e0 0x7fb6d4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 0x7fb6d419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d41121c0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d4112040 con 0x7fb6d4107fd0 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d4112340 con 0x7fb6d40690e0 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37864/0 (socket says 172.21.15.105:37864) 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 -- 172.21.15.105:0/1981596936 learned_addr learned my addr 172.21.15.105:0/1981596936 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:23.842 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6da372640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 0x7fb6d419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 -- 172.21.15.105:0/1981596936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6d40690e0 msgr2=0x7fb6d4198700 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6d40690e0 0x7fb6d4198700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 -- 172.21.15.105:0/1981596936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 msgr2=0x7fb6d419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 0x7fb6d419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 -- 172.21.15.105:0/1981596936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d419d6f0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6da372640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 0x7fb6d419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6d9370640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005940 tx=0x7fb6c402cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4043020 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6c402cca0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d419d980 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4003bf0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.841+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d41a5270 con 0x7fb6d4069aa0 2024-04-22T06:48:23.843 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.842+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d4068de0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.843+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb6c4046020 con 0x7fb6d4069aa0 2024-04-22T06:48:23.844 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.844+0000 7fb6caffd640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 0x7fb6c00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:23.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.844+0000 7fb6d9b71640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 0x7fb6c00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:23.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.844+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fb6c40f34c0 con 0x7fb6d4069aa0 2024-04-22T06:48:23.845 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.845+0000 7fb6d9b71640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 0x7fb6c00a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009670 tx=0x7fb6d0006bd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:23.846 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:23.848+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6c404d020 con 0x7fb6d4069aa0 2024-04-22T06:48:23.917 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:23 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/447063842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:48:23.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:23 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1734411580' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:48:23.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:23 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/447063842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:48:23.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:23 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1734411580' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:48:23.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:23 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/447063842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-22T06:48:23.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:23 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1734411580' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T06:48:24.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.069+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6d4199a50 con 0x7fb6d4069aa0 2024-04-22T06:48:24.067 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.069+0000 7fb6caffd640 1 -- 172.21.15.105:0/1981596936 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11956 (secure 0 0 0) 0x7fb6c40bf8b0 con 0x7fb6d4069aa0 2024-04-22T06:48:24.068 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:48:24.068 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:40:26.045831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":25691,"mds_2":25690,"mds_3":25685,"mds_4":16398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16398":{"gid":16398,"name":"g","rank":4,"incarnation":53,"state":"up:active","state_seq":18,"addr":"172.21.15.105:6839/798740274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":798740274},{"type":"v1","addr":"172.21.15.105:6839","nonce":798740274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16566":{"gid":16566,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/3795610143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":3795610143},{"type":"v1","addr":"172.21.15.105:6841","nonce":3795610143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25685":{"gid":25685,"name":"e","rank":3,"incarnation":51,"state":"up:active","state_seq":18,"addr":"172.21.15.112:6833/3361835468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":3361835468},{"type":"v1","addr":"172.21.15.112:6833","nonce":3361835468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25690":{"gid":25690,"name":"l","rank":2,"incarnation":55,"state":"up:active","state_seq":18,"addr":"172.21.15.177:6833/3918665016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3918665016},{"type":"v1","addr":"172.21.15.177:6833","nonce":3918665016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"k","rank":1,"incarnation":55,"state":"up:active","state_seq":18,"addr":"172.21.15.112:6835/1127626841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1127626841},{"type":"v1","addr":"172.21.15.112:6835","nonce":1127626841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/482071778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":482071778},{"type":"v1","addr":"172.21.15.112:6837","nonce":482071778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25828":{"gid":25828,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2359784785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2359784785},{"type":"v1","addr":"172.21.15.177:6837","nonce":2359784785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25834":{"gid":25834,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3491723308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3491723308},{"type":"v1","addr":"172.21.15.177:6839","nonce":3491723308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[25685,25691,16398,24500,25690]},"id":1}]} 2024-04-22T06:48:24.068 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 60 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 msgr2=0x7fb6c00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 0x7fb6c00a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009670 tx=0x7fb6d0006bd0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 msgr2=0x7fb6d4198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005940 tx=0x7fb6c402cab0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 shutdown_connections 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fb6c00a3da0 0x7fb6c00a6260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6d4107fd0 0x7fb6d419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.070 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d4069aa0 0x7fb6d4198c40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 --2- 172.21.15.105:0/1981596936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6d40690e0 0x7fb6d4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.071+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 >> 172.21.15.105:0/1981596936 conn(0x7fb6d4077e30 msgr2=0x7fb6d40fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.072+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 shutdown_connections 2024-04-22T06:48:24.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.072+0000 7fb6dbdd5640 1 -- 172.21.15.105:0/1981596936 wait complete. 2024-04-22T06:48:24.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 -- 172.21.15.105:0/55964156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa798101b60 msgr2=0x7fa79810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 --2- 172.21.15.105:0/55964156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa798101b60 0x7fa79810dc90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fa790009ca0 tx=0x7fa79002c160 comp rx=0 tx=0).stop 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 -- 172.21.15.105:0/55964156 shutdown_connections 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 --2- 172.21.15.105:0/55964156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa798101b60 0x7fa79810dc90 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 --2- 172.21.15.105:0/55964156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7981011a0 0x7fa798101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 --2- 172.21.15.105:0/55964156 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa798105ee0 0x7fa7981062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 -- 172.21.15.105:0/55964156 >> 172.21.15.105:0/55964156 conn(0x7fa798077c20 msgr2=0x7fa798078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 -- 172.21.15.105:0/55964156 shutdown_connections 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.181+0000 7fa79de06640 1 -- 172.21.15.105:0/55964156 wait complete. 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 Processor -- start 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 -- start start 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa798101b60 0x7fa798198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 0x7fa79819cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa798112270 con 0x7fa7981011a0 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981120f0 con 0x7fa798105ee0 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fa7981123f0 con 0x7fa798101b60 2024-04-22T06:48:24.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37892/0 (socket says 172.21.15.105:37892) 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 -- 172.21.15.105:0/1351213082 learned_addr learned my addr 172.21.15.105:0/1351213082 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa797fff640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 0x7fa79819cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 -- 172.21.15.105:0/1351213082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa798101b60 msgr2=0x7fa798198c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa798101b60 0x7fa798198c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 -- 172.21.15.105:0/1351213082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 msgr2=0x7fa79819cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 0x7fa79819cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 -- 172.21.15.105:0/1351213082 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79819d720 con 0x7fa7981011a0 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa797fff640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 0x7fa79819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:24.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7977fe640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b450 tx=0x7fa78800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788015020 con 0x7fa7981011a0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa788011d40 con 0x7fa7981011a0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79819da10 con 0x7fa7981011a0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.182+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78800f340 con 0x7fa7981011a0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.183+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7981a52a0 con 0x7fa7981011a0 2024-04-22T06:48:24.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.183+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa798069230 con 0x7fa7981011a0 2024-04-22T06:48:24.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.185+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fa788018020 con 0x7fa7981011a0 2024-04-22T06:48:24.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.185+0000 7fa7957fa640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 0x7fa7840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.185+0000 7fa78ffff640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 0x7fa7840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.185+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fa7880d0b60 con 0x7fa7981011a0 2024-04-22T06:48:24.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.186+0000 7fa78ffff640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 0x7fa7840a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa798105a70 tx=0x7fa78000b370 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.189+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa78801f020 con 0x7fa7981011a0 2024-04-22T06:48:24.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.402+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa798199a80 con 0x7fa7981011a0 2024-04-22T06:48:24.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.403+0000 7fa7957fa640 1 -- 172.21.15.105:0/1351213082 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+11956 (secure 0 0 0) 0x7fa798199a80 con 0x7fa7981011a0 2024-04-22T06:48:24.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:48:24.401 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/3495232000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":3495232000},{"type":"v1","addr":"172.21.15.105:6835","nonce":3495232000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2718881173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2718881173},{"type":"v1","addr":"172.21.15.105:6837","nonce":2718881173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":60,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-22T06:35:40.134334+0000","modified":"2024-04-22T06:40:26.045831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":25691,"mds_2":25690,"mds_3":25685,"mds_4":16398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16398":{"gid":16398,"name":"g","rank":4,"incarnation":53,"state":"up:active","state_seq":18,"addr":"172.21.15.105:6839/798740274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":798740274},{"type":"v1","addr":"172.21.15.105:6839","nonce":798740274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_16566":{"gid":16566,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6841/3795610143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6840","nonce":3795610143},{"type":"v1","addr":"172.21.15.105:6841","nonce":3795610143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6835/3449982583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6834","nonce":3449982583},{"type":"v1","addr":"172.21.15.177:6835","nonce":3449982583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.112:6839/4112363218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":4112363218},{"type":"v1","addr":"172.21.15.112:6839","nonce":4112363218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25685":{"gid":25685,"name":"e","rank":3,"incarnation":51,"state":"up:active","state_seq":18,"addr":"172.21.15.112:6833/3361835468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":3361835468},{"type":"v1","addr":"172.21.15.112:6833","nonce":3361835468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25690":{"gid":25690,"name":"l","rank":2,"incarnation":55,"state":"up:active","state_seq":18,"addr":"172.21.15.177:6833/3918665016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6832","nonce":3918665016},{"type":"v1","addr":"172.21.15.177:6833","nonce":3918665016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25691":{"gid":25691,"name":"k","rank":1,"incarnation":55,"state":"up:active","state_seq":18,"addr":"172.21.15.112:6835/1127626841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1127626841},{"type":"v1","addr":"172.21.15.112:6835","nonce":1127626841}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25817":{"gid":25817,"name":"h","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/482071778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":482071778},{"type":"v1","addr":"172.21.15.112:6837","nonce":482071778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25828":{"gid":25828,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6837/2359784785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6836","nonce":2359784785},{"type":"v1","addr":"172.21.15.177:6837","nonce":2359784785}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25834":{"gid":25834,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.177:6839/3491723308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6838","nonce":3491723308},{"type":"v1","addr":"172.21.15.177:6839","nonce":3491723308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[25685,25691,16398,24500,25690]},"id":1}]} 2024-04-22T06:48:24.401 INFO:teuthology.orchestra.run.smithi105.stderr:dumped fsmap epoch 60 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 msgr2=0x7fa7840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 0x7fa7840a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa798105a70 tx=0x7fa78000b370 comp rx=0 tx=0).stop 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 msgr2=0x7fa798198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fa78800b450 tx=0x7fa78800b910 comp rx=0 tx=0).stop 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 shutdown_connections 2024-04-22T06:48:24.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fa7840a3d20 0x7fa7840a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa798105ee0 0x7fa79819cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa798101b60 0x7fa798198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 --2- 172.21.15.105:0/1351213082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7981011a0 0x7fa798198730 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 >> 172.21.15.105:0/1351213082 conn(0x7fa798077c20 msgr2=0x7fa7980fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 shutdown_connections 2024-04-22T06:48:24.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.405+0000 7fa79de06640 1 -- 172.21.15.105:0/1351213082 wait complete. 2024-04-22T06:48:24.414 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:48:24.517 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2718862302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e41011a0 msgr2=0x7ff7e4101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2718862302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e41011a0 0x7ff7e4101620 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009930 tx=0x7ff7cc02bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2718862302 shutdown_connections 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2718862302 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e4101b60 0x7ff7e410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2718862302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e41011a0 0x7ff7e4101620 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2718862302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e4106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2718862302 >> 172.21.15.105:0/2718862302 conn(0x7ff7e4077e30 msgr2=0x7ff7e4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2718862302 shutdown_connections 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2718862302 wait complete. 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 Processor -- start 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- start start 2024-04-22T06:48:24.518 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 0x7ff7e4197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e4111160 con 0x7ff7e4101b60 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e4110fe0 con 0x7ff7e41011a0 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.514+0000 7ff7e93ce640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e41112e0 con 0x7ff7e4105e30 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55928/0 (socket says 172.21.15.105:55928) 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 -- 172.21.15.105:0/2567164429 learned_addr learned my addr 172.21.15.105:0/2567164429 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e2ffd640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 0x7ff7e4197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e37fe640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 -- 172.21.15.105:0/2567164429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 msgr2=0x7ff7e419bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e419bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 -- 172.21.15.105:0/2567164429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 msgr2=0x7ff7e4197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 0x7ff7e4197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 -- 172.21.15.105:0/2567164429 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e419c6f0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e2ffd640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 0x7ff7e4197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e37fe640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7dbfff640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005940 tx=0x7ff7cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0044e0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7cc0035f0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e419c980 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0476e0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.515+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e40690e0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.516+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e4102570 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.517+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7ff7cc003aa0 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.518+0000 7ff7e0ff9640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 0x7ff7d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.518+0000 7ff7e2ffd640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 0x7ff7d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.518+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7ff7cc0f4f40 con 0x7ff7e4101b60 2024-04-22T06:48:24.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.518+0000 7ff7e2ffd640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 0x7ff7d00a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4006fd0 tx=0x7ff7d4008040 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.522+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7cc0c12d0 con 0x7ff7e4101b60 2024-04-22T06:48:24.667 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:24 smithi105 ceph-mon[27916]: pgmap v348: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:48:24.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:24 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1981596936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:24.668 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:24 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/1351213082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:24.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.703+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7e4069b60 con 0x7ff7e4101b60 2024-04-22T06:48:24.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.704+0000 7ff7e0ff9640 1 -- 172.21.15.105:0/2567164429 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+24654 (secure 0 0 0) 0x7ff7cc0c0bd0 con 0x7ff7e4101b60 2024-04-22T06:48:24.703 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-22T06:48:24.703 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":114,"fsid":"48924e24-0071-11ef-bc93-c7b262605968","created":"2024-04-22T06:27:51.041097+0000","modified":"2024-04-22T06:48:16.161139+0000","last_up_change":"2024-04-22T06:34:50.673180+0000","last_in_change":"2024-04-22T06:34:31.020172+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:31:10.628361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-22T06:35:38.535076+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-22T06:35:39.107386+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":113,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a4d5ec0-d76f-4b44-9fcc-01b111b51a17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6803","nonce":1950705877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6805","nonce":1950705877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6809","nonce":1950705877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1950705877},{"type":"v1","addr":"172.21.15.105:6807","nonce":1950705877}]},"public_addr":"172.21.15.105:6803/1950705877","cluster_addr":"172.21.15.105:6805/1950705877","heartbeat_back_addr":"172.21.15.105:6809/1950705877","heartbeat_front_addr":"172.21.15.105:6807/1950705877","state":["exists","up"]},{"osd":1,"uuid":"e4ded2dc-2b79-4cd1-a34a-82778a1120b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6801","nonce":3644741495}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6803","nonce":3644741495}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6807","nonce":3644741495}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3644741495},{"type":"v1","addr":"172.21.15.112:6805","nonce":3644741495}]},"public_addr":"172.21.15.112:6801/3644741495","cluster_addr":"172.21.15.112:6803/3644741495","heartbeat_back_addr":"172.21.15.112:6807/3644741495","heartbeat_front_addr":"172.21.15.112:6805/3644741495","state":["exists","up"]},{"osd":2,"uuid":"9d48dd22-4dc5-4b82-bb01-119c1777318c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6801","nonce":1615473709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6803","nonce":1615473709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6807","nonce":1615473709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1615473709},{"type":"v1","addr":"172.21.15.177:6805","nonce":1615473709}]},"public_addr":"172.21.15.177:6801/1615473709","cluster_addr":"172.21.15.177:6803/1615473709","heartbeat_back_addr":"172.21.15.177:6807/1615473709","heartbeat_front_addr":"172.21.15.177:6805/1615473709","state":["exists","up"]},{"osd":3,"uuid":"b765227c-f85b-48e2-bc46-0de059180664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6811","nonce":3727777768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6813","nonce":3727777768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6817","nonce":3727777768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3727777768},{"type":"v1","addr":"172.21.15.105:6815","nonce":3727777768}]},"public_addr":"172.21.15.105:6811/3727777768","cluster_addr":"172.21.15.105:6813/3727777768","heartbeat_back_addr":"172.21.15.105:6817/3727777768","heartbeat_front_addr":"172.21.15.105:6815/3727777768","state":["exists","up"]},{"osd":4,"uuid":"1ae3793c-2262-486a-8754-ab46bcea415d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6809","nonce":2758455817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6811","nonce":2758455817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6815","nonce":2758455817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":2758455817},{"type":"v1","addr":"172.21.15.112:6813","nonce":2758455817}]},"public_addr":"172.21.15.112:6809/2758455817","cluster_addr":"172.21.15.112:6811/2758455817","heartbeat_back_addr":"172.21.15.112:6815/2758455817","heartbeat_front_addr":"172.21.15.112:6813/2758455817","state":["exists","up"]},{"osd":5,"uuid":"6f9fd1a3-a0ef-4d21-afed-1aca4e529af0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6809","nonce":1059606391}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6811","nonce":1059606391}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6815","nonce":1059606391}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":1059606391},{"type":"v1","addr":"172.21.15.177:6813","nonce":1059606391}]},"public_addr":"172.21.15.177:6809/1059606391","cluster_addr":"172.21.15.177:6811/1059606391","heartbeat_back_addr":"172.21.15.177:6815/1059606391","heartbeat_front_addr":"172.21.15.177:6813/1059606391","state":["exists","up"]},{"osd":6,"uuid":"2b6ab8f3-9288-43b7-886b-ce713b206fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6819","nonce":546897953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6821","nonce":546897953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6825","nonce":546897953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":546897953},{"type":"v1","addr":"172.21.15.105:6823","nonce":546897953}]},"public_addr":"172.21.15.105:6819/546897953","cluster_addr":"172.21.15.105:6821/546897953","heartbeat_back_addr":"172.21.15.105:6825/546897953","heartbeat_front_addr":"172.21.15.105:6823/546897953","state":["exists","up"]},{"osd":7,"uuid":"1d324cb7-ac5a-44a5-b33f-4fb140559f77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6817","nonce":1044643944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6819","nonce":1044643944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6823","nonce":1044643944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":1044643944},{"type":"v1","addr":"172.21.15.112:6821","nonce":1044643944}]},"public_addr":"172.21.15.112:6817/1044643944","cluster_addr":"172.21.15.112:6819/1044643944","heartbeat_back_addr":"172.21.15.112:6823/1044643944","heartbeat_front_addr":"172.21.15.112:6821/1044643944","state":["exists","up"]},{"osd":8,"uuid":"6c0422b5-87fa-4bc1-bb2c-1ef2eb25d6ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6817","nonce":76921589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6819","nonce":76921589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6823","nonce":76921589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":76921589},{"type":"v1","addr":"172.21.15.177:6821","nonce":76921589}]},"public_addr":"172.21.15.177:6817/76921589","cluster_addr":"172.21.15.177:6819/76921589","heartbeat_back_addr":"172.21.15.177:6823/76921589","heartbeat_front_addr":"172.21.15.177:6821/76921589","state":["exists","up"]},{"osd":9,"uuid":"0ed862a3-bff9-49d5-abfe-ff784bbad0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6827","nonce":716531130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6829","nonce":716531130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6833","nonce":716531130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":716531130},{"type":"v1","addr":"172.21.15.105:6831","nonce":716531130}]},"public_addr":"172.21.15.105:6827/716531130","cluster_addr":"172.21.15.105:6829/716531130","heartbeat_back_addr":"172.21.15.105:6833/716531130","heartbeat_front_addr":"172.21.15.105:6831/716531130","state":["exists","up"]},{"osd":10,"uuid":"8cd8511d-79e2-438b-8e64-d3936ea4fc3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6825","nonce":2010680205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6827","nonce":2010680205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6831","nonce":2010680205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":2010680205},{"type":"v1","addr":"172.21.15.112:6829","nonce":2010680205}]},"public_addr":"172.21.15.112:6825/2010680205","cluster_addr":"172.21.15.112:6827/2010680205","heartbeat_back_addr":"172.21.15.112:6831/2010680205","heartbeat_front_addr":"172.21.15.112:6829/2010680205","state":["exists","up"]},{"osd":11,"uuid":"747a1047-9993-4f2b-9650-a2cddcbe04ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6825","nonce":1875698423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6827","nonce":1875698423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6831","nonce":1875698423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":1875698423},{"type":"v1","addr":"172.21.15.177:6829","nonce":1875698423}]},"public_addr":"172.21.15.177:6825/1875698423","cluster_addr":"172.21.15.177:6827/1875698423","heartbeat_back_addr":"172.21.15.177:6831/1875698423","heartbeat_front_addr":"172.21.15.177:6829/1875698423","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:19.802260+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:30:44.381658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:07.614435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:32.370695+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:31:57.336554+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:21.283767+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:32:46.037340+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:10.944606+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:34.697320+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:33:59.524971+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:24.567046+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T06:34:48.839292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.177:6837/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6839/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.177:6838/3064875873":"2024-04-23T06:40:22.014751+0000","172.21.15.105:0/3300341732":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/1014082495":"2024-04-23T06:37:36.925007+0000","172.21.15.112:6836/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.105:6801/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6800/2556874834":"2024-04-23T06:37:36.925007+0000","172.21.15.177:6837/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.105:6801/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/817153020":"2024-04-23T06:37:36.925007+0000","172.21.15.105:0/937171571":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/768203659":"2024-04-23T06:28:17.391954+0000","172.21.15.177:6832/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6833/583171810":"2024-04-23T06:39:27.429757+0000","172.21.15.105:0/3406296347":"2024-04-23T06:28:44.539275+0000","172.21.15.105:6800/3871303088":"2024-04-23T06:28:44.539275+0000","172.21.15.105:0/2711052846":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1361172178":"2024-04-23T06:28:17.391954+0000","172.21.15.112:6836/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.112:6837/1826997924":"2024-04-23T06:40:20.987927+0000","172.21.15.112:6837/3970644798":"2024-04-23T06:38:36.718748+0000","172.21.15.105:6840/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6801/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/3570614146":"2024-04-23T06:37:36.925007+0000","172.21.15.105:6841/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6841/2147123680":"2024-04-23T06:38:36.825169+0000","172.21.15.105:6838/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6838/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.177:6839/1381266768":"2024-04-23T06:38:37.426711+0000","172.21.15.105:6840/85174468":"2024-04-23T06:40:21.258556+0000","172.21.15.105:6839/256609249":"2024-04-23T06:39:26.670520+0000","172.21.15.177:6836/678281724":"2024-04-23T06:40:22.020245+0000","172.21.15.177:6833/624977252":"2024-04-23T06:39:26.466860+0000","172.21.15.112:6834/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.105:6800/2332170576":"2024-04-23T06:28:17.391954+0000","172.21.15.105:0/1118894802":"2024-04-23T06:28:44.539275+0000","172.21.15.112:6835/3883368202":"2024-04-23T06:39:27.201039+0000","172.21.15.177:6836/4114728122":"2024-04-23T06:38:36.724579+0000","172.21.15.112:6832/583171810":"2024-04-23T06:39:27.429757+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.706+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 msgr2=0x7ff7d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.706+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 0x7ff7d00a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4006fd0 tx=0x7ff7d4008040 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 msgr2=0x7ff7e4197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc005940 tx=0x7ff7cc0042f0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 shutdown_connections 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7ff7d00a3d20 0x7ff7d00a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff7e4105e30 0x7ff7e419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7e4101b60 0x7ff7e4197c40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 --2- 172.21.15.105:0/2567164429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7e41011a0 0x7ff7e4197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 >> 172.21.15.105:0/2567164429 conn(0x7ff7e4077e30 msgr2=0x7ff7e40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.705 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 shutdown_connections 2024-04-22T06:48:24.706 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.707+0000 7ff7e93ce640 1 -- 172.21.15.105:0/2567164429 wait complete. 2024-04-22T06:48:24.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 -- 172.21.15.105:0/40259756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101a40 msgr2=0x7fcb9c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/40259756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101a40 0x7fcb9c105a80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009900 tx=0x7fcb9002bc50 comp rx=0 tx=0).stop 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 -- 172.21.15.105:0/40259756 shutdown_connections 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/40259756 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 0x7fcb9c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/40259756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101a40 0x7fcb9c105a80 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/40259756 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb9c101090 0x7fcb9c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 -- 172.21.15.105:0/40259756 >> 172.21.15.105:0/40259756 conn(0x7fcb9c077f50 msgr2=0x7fcb9c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 -- 172.21.15.105:0/40259756 shutdown_connections 2024-04-22T06:48:24.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.821+0000 7fcba1ca8640 1 -- 172.21.15.105:0/40259756 wait complete. 2024-04-22T06:48:24.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 Processor -- start 2024-04-22T06:48:24.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 -- start start 2024-04-22T06:48:24.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 0x7fcb9c19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb9c1985d0 0x7fcb9c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c111120 con 0x7fcb9c101090 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c110fa0 con 0x7fcb9c106160 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcba1ca8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9c1112a0 con 0x7fcb9c1985d0 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcb9b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcb9b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55952/0 (socket says 172.21.15.105:55952) 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.822+0000 7fcb9b7fe640 1 -- 172.21.15.105:0/2465886982 learned_addr learned my addr 172.21.15.105:0/2465886982 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9affd640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 0x7fcb9c19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 -- 172.21.15.105:0/2465886982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb9c1985d0 msgr2=0x7fcb9c198a80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb9c1985d0 0x7fcb9c198a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 -- 172.21.15.105:0/2465886982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 msgr2=0x7fcb9c19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 0x7fcb9c19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 -- 172.21.15.105:0/2465886982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9c1990f0 con 0x7fcb9c101090 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb9b7fe640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00b450 tx=0x7fcb8c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c015020 con 0x7fcb9c101090 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9c1993e0 con 0x7fcb9c101090 2024-04-22T06:48:24.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c011d40 con 0x7fcb9c101090 2024-04-22T06:48:24.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9c1a5680 con 0x7fcb9c101090 2024-04-22T06:48:24.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.823+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c01a400 con 0x7fcb9c101090 2024-04-22T06:48:24.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.824+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb9c068d70 con 0x7fcb9c101090 2024-04-22T06:48:24.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.825+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fcb8c018020 con 0x7fcb9c101090 2024-04-22T06:48:24.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.826+0000 7fcb98ff9640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 0x7fcb840a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:24.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.826+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9487+0+0 (secure 0 0 0) 0x7fcb8c0d05e0 con 0x7fcb9c101090 2024-04-22T06:48:24.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.826+0000 7fcb9affd640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 0x7fcb840a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:24.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.826+0000 7fcb9affd640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 0x7fcb840a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb90004120 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:24.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:24.830+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb8c01f020 con 0x7fcb9c101090 2024-04-22T06:48:24.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:24 smithi177 ceph-mon[31346]: pgmap v348: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:48:24.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:24 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1981596936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:24.944 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:24 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/1351213082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:24.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:24 smithi112 ceph-mon[31236]: pgmap v348: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-22T06:48:24.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:24 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1981596936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:24.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:24 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/1351213082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-22T06:48:25.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.047+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcb9c199b00 con 0x7fcb9c101090 2024-04-22T06:48:25.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.051+0000 7fcb98ff9640 1 -- 172.21.15.105:0/2465886982 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) ==== 161+0+0 (secure 0 0 0) 0x7fcb8c00d040 con 0x7fcb9c101090 2024-04-22T06:48:25.049 INFO:teuthology.orchestra.run.smithi105.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 msgr2=0x7fcb840a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 0x7fcb840a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcb90006010 tx=0x7fcb90004120 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 msgr2=0x7fcb9c19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c00b450 tx=0x7fcb8c00b910 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 shutdown_connections 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.054+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fcb840a3d20 0x7fcb840a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb9c1985d0 0x7fcb9c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb9c106160 0x7fcb9c19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 --2- 172.21.15.105:0/2465886982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb9c101090 0x7fcb9c19e420 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 >> 172.21.15.105:0/2465886982 conn(0x7fcb9c077f50 msgr2=0x7fcb9c069440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 shutdown_connections 2024-04-22T06:48:25.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.055+0000 7fcba1ca8640 1 -- 172.21.15.105:0/2465886982 wait complete. 2024-04-22T06:48:25.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-22T06:48:25.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 -- 172.21.15.105:0/2843897995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0106f30 msgr2=0x7fbea0107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 --2- 172.21.15.105:0/2843897995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0106f30 0x7fbea0107310 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fbe98009930 tx=0x7fbe9802bc30 comp rx=0 tx=0).stop 2024-04-22T06:48:25.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 -- 172.21.15.105:0/2843897995 shutdown_connections 2024-04-22T06:48:25.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 --2- 172.21.15.105:0/2843897995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100e10 0x7fbea010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 --2- 172.21.15.105:0/2843897995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0100450 0x7fbea01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 --2- 172.21.15.105:0/2843897995 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0106f30 0x7fbea0107310 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 -- 172.21.15.105:0/2843897995 >> 172.21.15.105:0/2843897995 conn(0x7fbea0077410 msgr2=0x7fbea00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 -- 172.21.15.105:0/2843897995 shutdown_connections 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 -- 172.21.15.105:0/2843897995 wait complete. 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.174+0000 7fbea83f4640 1 Processor -- start 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 -- start start 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 0x7fbea019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0106f30 0x7fbea01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbea0112290 con 0x7fbea0100e10 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbea0112110 con 0x7fbea0106f30 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7fbea0112410 con 0x7fbea0100450 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56016/0 (socket says 172.21.15.105:56016) 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 -- 172.21.15.105:0/575852174 learned_addr learned my addr 172.21.15.105:0/575852174 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea6190640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 0x7fbea019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea6991640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0106f30 0x7fbea01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 -- 172.21.15.105:0/575852174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 msgr2=0x7fbea019e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 0x7fbea019e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 -- 172.21.15.105:0/575852174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0106f30 msgr2=0x7fbea01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0106f30 0x7fbea01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 -- 172.21.15.105:0/575852174 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea0198d30 con 0x7fbea0100e10 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea6190640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 0x7fbea019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea598f640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400ebc0 tx=0x7fbe9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94010630 con 0x7fbea0100e10 2024-04-22T06:48:25.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe94010c80 con 0x7fbea0100e10 2024-04-22T06:48:25.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea0199020 con 0x7fbea0100e10 2024-04-22T06:48:25.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.175+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94018b60 con 0x7fbea0100e10 2024-04-22T06:48:25.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.176+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea01a52c0 con 0x7fbea0100e10 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.176+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea00691c0 con 0x7fbea0100e10 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.177+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fbe94018d00 con 0x7fbea0100e10 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.178+0000 7fbe937fe640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 0x7fbe8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.178+0000 7fbea6190640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 0x7fbe8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.178+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9873+0+0 (secure 0 0 0) 0x7fbe940d1770 con 0x7fbea0100e10 2024-04-22T06:48:25.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.179+0000 7fbea6190640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 0x7fbe8c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe98005ab0 tx=0x7fbe98005fd0 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:25.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.182+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe9409d270 con 0x7fbea0100e10 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mds[66023]: -- 172.21.15.177:0/3667405797 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56417306da40 con 0x564173041000 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: starting mds.c at 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mds[61302]: -- 172.21.15.177:0/3460950788 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa49cf5a40 con 0x55aa49cc8800 2024-04-22T06:48:25.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: starting mds.f at 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mds[66044]: -- 172.21.15.177:0/4096961917 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc04c2fa40 con 0x55dc04c02800 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: starting mds.i at 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.241 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mds[65493]: -- 172.21.15.177:0/2023148951 <== mon.1 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a39546fa40 con 0x55a395442800 2024-04-22T06:48:25.242 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: starting mds.l at 2024-04-22T06:48:25.393 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:25.395+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbea010a410 con 0x7fbea0100e10 2024-04-22T06:48:25.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a[27912]: 2024-04-22T06:48:25.048+0000 7f71fc197700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:48:25.418 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.418 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.418 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mds[88422]: -- 172.21.15.105:0/3187545200 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645452b3a40 con 0x564545287000 2024-04-22T06:48:25.418 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: starting mds.a at 2024-04-22T06:48:25.419 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.419 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.419 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mds[87524]: -- 172.21.15.105:0/2216660984 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebcf775a40 con 0x55ebcf748c00 2024-04-22T06:48:25.419 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: starting mds.g at 2024-04-22T06:48:25.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: starting mds.e at 2024-04-22T06:48:25.458 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.458 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mds[64863]: -- 172.21.15.112:0/3205590429 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0aa24fa40 con 0x55f0aa222c00 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: starting mds.h at 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mds[64472]: -- 172.21.15.112:0/428942136 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a42db084e0 con 0x55a42dade800 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mds[64472]: -- 172.21.15.112:0/428942136 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a42db0ba40 con 0x55a42dade800 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: starting mds.k at 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: ignoring --setuser ceph since I am not root 2024-04-22T06:48:25.459 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: ignoring --setgroup ceph since I am not root 2024-04-22T06:48:25.460 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mds[61057]: -- 172.21.15.112:0/2368104246 <== mon.0 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562aac3cba40 con 0x562aac39ec00 2024-04-22T06:48:25.460 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: starting mds.b at 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2567164429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:48:25.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T06:48:25.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:25 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2567164429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:48:25.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T06:48:25.959 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:25 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2567164429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/2465886982' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: osdmap e115: 12 total, 12 up, 12 in 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-22T06:48:25.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:25 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T06:48:26.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.057+0000 7fbe937fe640 1 -- 172.21.15.105:0/575852174 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) ==== 104+0+0 (secure 0 0 0) 0x7fbe940a1570 con 0x7fbea0100e10 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 msgr2=0x7fbe8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 0x7fbe8c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe98005ab0 tx=0x7fbe98005fd0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 msgr2=0x7fbea019e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400ebc0 tx=0x7fbe9400c6a0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 shutdown_connections 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7fbe8c0a3d20 0x7fbe8c0a61e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbea0106f30 0x7fbea01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbea0100e10 0x7fbea019e940 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 --2- 172.21.15.105:0/575852174 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbea0100450 0x7fbea019e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 >> 172.21.15.105:0/575852174 conn(0x7fbea0077410 msgr2=0x7fbea0105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 shutdown_connections 2024-04-22T06:48:26.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.061+0000 7fbea83f4640 1 -- 172.21.15.105:0/575852174 wait complete. 2024-04-22T06:48:26.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 -- 172.21.15.105:0/2873017736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc4107270 msgr2=0x7f9bc410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/2873017736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc4107270 0x7f9bc410ee40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009ca0 tx=0x7f9bbc02c0f0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 -- 172.21.15.105:0/2873017736 shutdown_connections 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/2873017736 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc4107270 0x7f9bc410ee40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/2873017736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc4102b50 0x7f9bc4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/2873017736 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc41021a0 0x7f9bc4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 -- 172.21.15.105:0/2873017736 >> 172.21.15.105:0/2873017736 conn(0x7f9bc40793e0 msgr2=0x7f9bc40797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 -- 172.21.15.105:0/2873017736 shutdown_connections 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.188+0000 7f9bca2e4640 1 -- 172.21.15.105:0/2873017736 wait complete. 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 Processor -- start 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 -- start start 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 0x7f9bc4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc419cfc0 0x7f9bc419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41121e0 con 0x7f9bc41021a0 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4112060 con 0x7f9bc419cfc0 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bca2e4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4112360 con 0x7f9bc4107270 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56034/0 (socket says 172.21.15.105:56034) 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 -- 172.21.15.105:0/4279161305 learned_addr learned my addr 172.21.15.105:0/4279161305 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc8881640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc419cfc0 0x7f9bc419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc37fe640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 0x7f9bc4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 -- 172.21.15.105:0/4279161305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 msgr2=0x7f9bc4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 0x7f9bc4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 -- 172.21.15.105:0/4279161305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc419cfc0 msgr2=0x7f9bc419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc419cfc0 0x7f9bc419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 -- 172.21.15.105:0/4279161305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc419dab0 con 0x7f9bc41021a0 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc37fe640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 0x7f9bc4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.189+0000 7f9bc3fff640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400e9f0 tx=0x7f9bb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.190+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400cd80 con 0x7f9bc41021a0 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.190+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bb40054d0 con 0x7f9bc41021a0 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.190+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc419dda0 con 0x7f9bc41021a0 2024-04-22T06:48:26.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.190+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4010690 con 0x7f9bc41021a0 2024-04-22T06:48:26.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.190+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc41a5540 con 0x7f9bc41021a0 2024-04-22T06:48:26.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.191+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc40691c0 con 0x7f9bc41021a0 2024-04-22T06:48:26.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.192+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f9bb4004e90 con 0x7f9bc41021a0 2024-04-22T06:48:26.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.192+0000 7f9bc17fa640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 0x7f9bac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:26.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.192+0000 7f9bc37fe640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 0x7f9bac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:26.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.192+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9873+0+0 (secure 0 0 0) 0x7f9bb40d1200 con 0x7f9bc41021a0 2024-04-22T06:48:26.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.193+0000 7f9bc37fe640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 0x7f9bac0a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005e00 tx=0x7f9bb8005c70 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:26.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.197+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9bb409d350 con 0x7f9bc41021a0 2024-04-22T06:48:26.371 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:26.373+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9bc4199a50 con 0x7f9bc41021a0 2024-04-22T06:48:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: pgmap v350: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 114 B/s wr, 11 op/s 2024-04-22T06:48:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:26.918 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: Cluster is now healthy 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6838/3319092928,v1:172.21.15.105:6839/3319092928] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6833/3999894286,v1:172.21.15.112:6835/3999894286] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6832/2356389046,v1:172.21.15.112:6834/2356389046] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6833/3082211470,v1:172.21.15.177:6835/3082211470] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6832/1803234154,v1:172.21.15.177:6834/1803234154] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.105:6840/1601300018,v1:172.21.15.105:6841/1601300018] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6836/4255209101,v1:172.21.15.177:6838/4255209101] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.177:6837/3676150380,v1:172.21.15.177:6839/3676150380] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6836/67235106,v1:172.21.15.112:6837/67235106] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: mds.? [v2:172.21.15.112:6838/678770889,v1:172.21.15.112:6839/678770889] up:boot 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: fsmap 12 up:standby 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:48:26.919 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:48:26.920 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:26 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:26.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: pgmap v350: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 114 B/s wr, 11 op/s 2024-04-22T06:48:26.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:26.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:48:26.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: Cluster is now healthy 2024-04-22T06:48:26.962 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6838/3319092928,v1:172.21.15.105:6839/3319092928] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6833/3999894286,v1:172.21.15.112:6835/3999894286] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6832/2356389046,v1:172.21.15.112:6834/2356389046] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6833/3082211470,v1:172.21.15.177:6835/3082211470] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6832/1803234154,v1:172.21.15.177:6834/1803234154] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.105:6840/1601300018,v1:172.21.15.105:6841/1601300018] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6836/4255209101,v1:172.21.15.177:6838/4255209101] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.177:6837/3676150380,v1:172.21.15.177:6839/3676150380] up:boot 2024-04-22T06:48:26.963 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6836/67235106,v1:172.21.15.112:6837/67235106] up:boot 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: mds.? [v2:172.21.15.112:6838/678770889,v1:172.21.15.112:6839/678770889] up:boot 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: fsmap 12 up:standby 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:48:26.964 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:26 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: pgmap v350: 97 pgs: 97 active+clean; 2.1 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 114 B/s wr, 11 op/s 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: Cluster is now healthy 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/575852174' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6838/3319092928,v1:172.21.15.105:6839/3319092928] up:boot 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6833/3999894286,v1:172.21.15.112:6835/3999894286] up:boot 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6832/2356389046,v1:172.21.15.112:6834/2356389046] up:boot 2024-04-22T06:48:26.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6833/3082211470,v1:172.21.15.177:6835/3082211470] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6832/1803234154,v1:172.21.15.177:6834/1803234154] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.105:6840/1601300018,v1:172.21.15.105:6841/1601300018] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6836/4255209101,v1:172.21.15.177:6838/4255209101] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.177:6837/3676150380,v1:172.21.15.177:6839/3676150380] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6836/67235106,v1:172.21.15.112:6837/67235106] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: mds.? [v2:172.21.15.112:6838/678770889,v1:172.21.15.112:6839/678770889] up:boot 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: fsmap 12 up:standby 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-22T06:48:26.991 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-22T06:48:26.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-22T06:48:26.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-22T06:48:26.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='mgr.15357 172.21.15.105:0/3336882255' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-22T06:48:26.992 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:26 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:27.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.058+0000 7f9bc17fa640 1 -- 172.21.15.105:0/4279161305 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) ==== 181+0+0 (secure 0 0 0) 0x7f9bb409cc50 con 0x7f9bc41021a0 2024-04-22T06:48:27.057 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'cephfs_metadata' removed 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.061+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 msgr2=0x7f9bac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.061+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 0x7f9bac0a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005e00 tx=0x7f9bb8005c70 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.061+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 msgr2=0x7f9bc4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.061+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400e9f0 tx=0x7f9bb400edf0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 shutdown_connections 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f9bac0a3d20 0x7f9bac0a61e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9bc419cfc0 0x7f9bc419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9bc4107270 0x7f9bc4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 --2- 172.21.15.105:0/4279161305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9bc41021a0 0x7f9bc4198700 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 >> 172.21.15.105:0/4279161305 conn(0x7f9bc40793e0 msgr2=0x7f9bc40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 shutdown_connections 2024-04-22T06:48:27.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.062+0000 7f9bca2e4640 1 -- 172.21.15.105:0/4279161305 wait complete. 2024-04-22T06:48:27.073 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-22T06:48:27.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 -- 172.21.15.105:0/1462363135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0101aa0 msgr2=0x7f04d0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/1462363135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0101aa0 0x7f04d0105a60 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009900 tx=0x7f04b802bc30 comp rx=0 tx=0).stop 2024-04-22T06:48:27.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 -- 172.21.15.105:0/1462363135 shutdown_connections 2024-04-22T06:48:27.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/1462363135 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d0106140 0x7f04d010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/1462363135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0101aa0 0x7f04d0105a60 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/1462363135 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01010f0 0x7f04d01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 -- 172.21.15.105:0/1462363135 >> 172.21.15.105:0/1462363135 conn(0x7f04d0076f60 msgr2=0x7f04d0077370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 -- 172.21.15.105:0/1462363135 shutdown_connections 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.181+0000 7f04d4b6d640 1 -- 172.21.15.105:0/1462363135 wait complete. 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 Processor -- start 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 -- start start 2024-04-22T06:48:27.183 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d01010f0 0x7f04d019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01985e0 0x7f04d0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f04d0112200 con 0x7f04d0106140 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f04d0112080 con 0x7f04d01010f0 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04d4b6d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 -- 0x7f04d0112380 con 0x7f04d01985e0 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:56064/0 (socket says 172.21.15.105:56064) 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 -- 172.21.15.105:0/4001863450 learned_addr learned my addr 172.21.15.105:0/4001863450 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-04-22T06:48:27.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04ced9d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01985e0 0x7f04d0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04ce59c640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d01010f0 0x7f04d019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 -- 172.21.15.105:0/4001863450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01985e0 msgr2=0x7f04d0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01985e0 0x7f04d0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 -- 172.21.15.105:0/4001863450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d01010f0 msgr2=0x7f04d019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d01010f0 0x7f04d019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 -- 172.21.15.105:0/4001863450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d0199100 con 0x7f04d0106140 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.182+0000 7f04c7fff640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f04b8036040 tx=0x7f04b802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.183+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8043020 con 0x7f04d0106140 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.183+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d0199390 con 0x7f04d0106140 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.183+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b802cbe0 con 0x7f04d0106140 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.183+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8003bf0 con 0x7f04d0106140 2024-04-22T06:48:27.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.183+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d01a55e0 con 0x7f04d0106140 2024-04-22T06:48:27.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.184+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d0102540 con 0x7f04d0106140 2024-04-22T06:48:27.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.185+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f04b8046020 con 0x7f04d0106140 2024-04-22T06:48:27.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.185+0000 7f04c77fe640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 0x7f04bc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-22T06:48:27.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.185+0000 7f04ce59c640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 0x7f04bc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-22T06:48:27.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.186+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9377+0+0 (secure 0 0 0) 0x7f04b80f2cf0 con 0x7f04d0106140 2024-04-22T06:48:27.187 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.186+0000 7f04ce59c640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 0x7f04bc0a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f04c0005fd0 tx=0x7f04c0005850 comp rx=0 tx=0).ready entity=mgr.15357 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-22T06:48:27.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.190+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04b80f7070 con 0x7f04d0106140 2024-04-22T06:48:27.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:27.365+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f04d0109430 con 0x7f04d0106140 2024-04-22T06:48:28.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.064+0000 7f04c77fe640 1 -- 172.21.15.105:0/4001863450 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) ==== 169+0+0 (secure 0 0 0) 0x7f04b804d020 con 0x7f04d0106140 2024-04-22T06:48:28.063 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'cephfs_data' removed 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.066+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 msgr2=0x7f04bc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.066+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 0x7f04bc0a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f04c0005fd0 tx=0x7f04c0005850 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.066+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 msgr2=0x7f04d019e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.066+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f04b8036040 tx=0x7f04b802c7e0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.066+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 shutdown_connections 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:6800/971331810,v1:172.21.15.105:6801/971331810] conn(0x7f04bc0a3d20 0x7f04bc0a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f04d01985e0 0x7f04d0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04d0106140 0x7f04d019e970 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 --2- 172.21.15.105:0/4001863450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04d01010f0 0x7f04d019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-22T06:48:28.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 >> 172.21.15.105:0/4001863450 conn(0x7f04d0076f60 msgr2=0x7f04d00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-22T06:48:28.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 shutdown_connections 2024-04-22T06:48:28.066 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-22T06:48:28.067+0000 7f04d4b6d640 1 -- 172.21.15.105:0/4001863450 wait complete. 2024-04-22T06:48:28.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:48:28.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:48:28.136 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:48:28.169 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T06:48:28.170 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:48:28.185 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:48:28.203 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T06:48:28.227 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T06:48:28.228 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T06:48:28.228 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a 2024-04-22T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-mon[27916]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-mon[27916]: pgmap v352: 65 pgs: 65 active+clean; 577 KiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-mon[27916]: from='client.? 172.21.15.105:0/4001863450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 systemd[1]: Stopping Ceph mon.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:48:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:28 smithi112 ceph-mon[31236]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:48:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:28 smithi112 ceph-mon[31236]: pgmap v352: 65 pgs: 65 active+clean; 577 KiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:48:28.458 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:28 smithi112 ceph-mon[31236]: from='client.? 172.21.15.105:0/4001863450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:28.489 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4279161305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-22T06:48:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:28 smithi177 ceph-mon[31346]: osdmap e116: 12 total, 12 up, 12 in 2024-04-22T06:48:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:28 smithi177 ceph-mon[31346]: pgmap v352: 65 pgs: 65 active+clean; 577 KiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-22T06:48:28.490 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:28 smithi177 ceph-mon[31346]: from='client.? 172.21.15.105:0/4001863450' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-22T06:48:28.776 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a[27912]: 2024-04-22T06:48:28.511+0000 7f72019a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:28.776 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a[27912]: 2024-04-22T06:48:28.511+0000 7f72019a2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T06:48:29.077 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:28 smithi105 podman[91534]: 2024-04-22 06:48:28.778017775 +0000 UTC m=+0.361174847 container died 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:48:29.386 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:29 smithi105 podman[91534]: 2024-04-22 06:48:29.187854655 +0000 UTC m=+0.771011786 container cleanup 0d1415368ca01eefc80df67f4413cebb61d07ae544c5e3685d04e179c8c912f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-22T06:48:29.386 INFO:journalctl@ceph.mon.a.smithi105.stdout:Apr 22 06:48:29 smithi105 bash[91534]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-a 2024-04-22T06:48:29.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T06:48:29.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:29.649 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T06:48:29.649 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-22T06:48:29.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.b 2024-04-22T06:48:29.958 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:29 smithi112 systemd[1]: Stopping Ceph mon.b for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:30.314 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-b[31231]: 2024-04-22T06:48:29.964+0000 7f4da227a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:30.314 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-b[31231]: 2024-04-22T06:48:29.964+0000 7f4da227a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:48:30.314 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:30 smithi112 podman[67038]: 2024-04-22 06:48:30.057649217 +0000 UTC m=+0.209967429 container died 8c86d0b36e3874fc7bd2a3d24120ce7c76fede91bf67425c1abff82ecd613c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:48:30.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:30 smithi112 podman[67038]: 2024-04-22 06:48:30.415929899 +0000 UTC m=+0.568248108 container cleanup 8c86d0b36e3874fc7bd2a3d24120ce7c76fede91bf67425c1abff82ecd613c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:48:30.708 INFO:journalctl@ceph.mon.b.smithi112.stdout:Apr 22 06:48:30 smithi112 bash[67038]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-b 2024-04-22T06:48:30.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T06:48:30.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:30.928 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-22T06:48:30.928 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-22T06:48:30.928 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.c 2024-04-22T06:48:31.211 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:30 smithi177 systemd[1]: Stopping Ceph mon.c for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:31.471 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c[31342]: 2024-04-22T06:48:31.239+0000 7f7a3caa0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:31.471 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c[31342]: 2024-04-22T06:48:31.239+0000 7f7a3caa0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T06:48:31.471 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 podman[68237]: 2024-04-22 06:48:31.382572599 +0000 UTC m=+0.250170589 container died f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:31.989 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 podman[68237]: 2024-04-22 06:48:31.723857191 +0000 UTC m=+0.591455178 container cleanup f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:48:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 bash[68237]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c 2024-04-22T06:48:31.990 INFO:journalctl@ceph.mon.c.smithi177.stdout:Apr 22 06:48:31 smithi177 podman[68252]: 2024-04-22 06:48:31.941748276 +0000 UTC m=+0.560016376 container remove f0e1fc0037807e80df55ac4ad39bc0b3ff37307da0b0b6a292ffd914ce859ac8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:48:32.200 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T06:48:32.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:32.242 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-22T06:48:32.242 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T06:48:32.242 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x 2024-04-22T06:48:32.523 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:32 smithi105 systemd[1]: Stopping Ceph mgr.x for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:32.799 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x[28147]: 2024-04-22T06:48:32.524+0000 7fb0c00c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:32.799 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:32 smithi105 podman[91659]: 2024-04-22 06:48:32.556313271 +0000 UTC m=+0.130244604 container died f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-22T06:48:33.082 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:32 smithi105 podman[91659]: 2024-04-22 06:48:32.908954249 +0000 UTC m=+0.482885588 container cleanup f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-22T06:48:33.083 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:32 smithi105 bash[91659]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x 2024-04-22T06:48:33.321 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T06:48:33.342 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:33 smithi105 podman[91673]: 2024-04-22 06:48:33.085373214 +0000 UTC m=+0.526185201 container remove f0a31be559b2d1e3998e85190e032d85897d2c9853ff82aaa7f075f43c602d44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:86f7587a5a09af35f0895e1a2d08527638fae697, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T06:48:33.342 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:33 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-22T06:48:33.342 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:33 smithi105 systemd[1]: Stopped Ceph mgr.x for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:48:33.342 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Apr 22 06:48:33 smithi105 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 21.963s CPU time. 2024-04-22T06:48:33.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:33.356 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T06:48:33.356 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T06:48:33.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.y 2024-04-22T06:48:33.652 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:48:33 smithi112 systemd[1]: Stopping Ceph mgr.y for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:33.958 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:48:33 smithi112 podman[67161]: 2024-04-22 06:48:33.743826639 +0000 UTC m=+0.196409733 container died ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:34.381 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:48:34 smithi112 podman[67161]: 2024-04-22 06:48:34.149171431 +0000 UTC m=+0.601754538 container cleanup ed5a0a9e12a66217ee25c5361aefc1b3a71872f5b0251441925e87c0fc1597aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:48:34.381 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Apr 22 06:48:34 smithi112 bash[67161]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-y 2024-04-22T06:48:34.521 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T06:48:34.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:34.557 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T06:48:34.557 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-22T06:48:34.557 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.z 2024-04-22T06:48:34.896 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:48:34 smithi177 systemd[1]: Stopping Ceph mgr.z for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:35.168 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:48:34 smithi177 podman[68370]: 2024-04-22 06:48:34.898699485 +0000 UTC m=+0.148329978 container died a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:48:35.434 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:48:35 smithi177 podman[68370]: 2024-04-22 06:48:35.244330633 +0000 UTC m=+0.493961120 container cleanup a04243b094b253e3e3a815e1d50c2af350ce1d1ac867efb524b3029be713eab9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:48:35.435 INFO:journalctl@ceph.mgr.z.smithi177.stdout:Apr 22 06:48:35 smithi177 bash[68370]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mgr-z 2024-04-22T06:48:35.659 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-22T06:48:35.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:35.695 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-22T06:48:35.695 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T06:48:35.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.0 2024-04-22T06:48:36.027 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:35 smithi105 systemd[1]: Stopping Ceph osd.0 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:36.418 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0[42796]: 2024-04-22T06:48:36.029+0000 7fb7dbdce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:36.418 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0[42796]: 2024-04-22T06:48:36.029+0000 7fb7dbdce700 -1 osd.0 117 *** Got signal Terminated *** 2024-04-22T06:48:36.418 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0[42796]: 2024-04-22T06:48:36.029+0000 7fb7dbdce700 -1 osd.0 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:48:41.779 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:41 smithi105 podman[91781]: 2024-04-22 06:48:41.512527687 +0000 UTC m=+5.620398405 container died c8a378707c294cc2e39dc236bb983b6f10770296404b164cd5686ed5a8a3b6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-22T06:48:42.167 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:41 smithi105 podman[91781]: 2024-04-22 06:48:41.880023896 +0000 UTC m=+5.987894611 container cleanup c8a378707c294cc2e39dc236bb983b6f10770296404b164cd5686ed5a8a3b6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T06:48:42.168 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:41 smithi105 bash[91781]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0 2024-04-22T06:48:42.539 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91798]: 2024-04-22 06:48:42.189665482 +0000 UTC m=+0.671420201 container remove c8a378707c294cc2e39dc236bb983b6f10770296404b164cd5686ed5a8a3b6b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-22T06:48:42.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91887]: 2024-04-22 06:48:42.444985901 +0000 UTC m=+0.018418422 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:48:42.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91887]: 2024-04-22 06:48:42.549033201 +0000 UTC m=+0.122465759 container create 3e2b9112eaa60b9f741184f19d0959a63be7a31bc4daf651a36bdafc7f045253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:42.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91887]: 2024-04-22 06:48:42.751594038 +0000 UTC m=+0.325026554 container init 3e2b9112eaa60b9f741184f19d0959a63be7a31bc4daf651a36bdafc7f045253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:42.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91887]: 2024-04-22 06:48:42.755094473 +0000 UTC m=+0.328526997 container start 3e2b9112eaa60b9f741184f19d0959a63be7a31bc4daf651a36bdafc7f045253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:48:42.849 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:42 smithi105 podman[91887]: 2024-04-22 06:48:42.851215581 +0000 UTC m=+0.424648105 container attach 3e2b9112eaa60b9f741184f19d0959a63be7a31bc4daf651a36bdafc7f045253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2) 2024-04-22T06:48:43.167 INFO:journalctl@ceph.osd.0.smithi105.stdout:Apr 22 06:48:43 smithi105 podman[91887]: 2024-04-22 06:48:43.040181401 +0000 UTC m=+0.613613913 container died 3e2b9112eaa60b9f741184f19d0959a63be7a31bc4daf651a36bdafc7f045253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T06:48:43.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T06:48:43.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:43.735 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T06:48:43.735 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T06:48:43.735 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.1 2024-04-22T06:48:44.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:43 smithi112 systemd[1]: Stopping Ceph osd.1 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:48:44.052+0000 7f64e32d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:48:44.052+0000 7f64e32d5700 -1 osd.1 117 *** Got signal Terminated *** 2024-04-22T06:48:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1[35863]: 2024-04-22T06:48:44.052+0000 7f64e32d5700 -1 osd.1 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:48:49.760 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:49 smithi112 podman[67297]: 2024-04-22 06:48:49.4815134 +0000 UTC m=+5.552012477 container died 7a0008614c639d74159c68c7dcd10e99d2184463c45dd07d154ee1bb5bfa3ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T06:48:50.135 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:49 smithi112 podman[67297]: 2024-04-22 06:48:49.84470292 +0000 UTC m=+5.915202006 container cleanup 7a0008614c639d74159c68c7dcd10e99d2184463c45dd07d154ee1bb5bfa3ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:48:50.135 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:49 smithi112 bash[67297]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1 2024-04-22T06:48:50.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67314]: 2024-04-22 06:48:50.137557989 +0000 UTC m=+0.648459616 container remove 7a0008614c639d74159c68c7dcd10e99d2184463c45dd07d154ee1bb5bfa3ab0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T06:48:50.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.342959662 +0000 UTC m=+0.018137728 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:48:50.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.472069249 +0000 UTC m=+0.147247310 container create 9360e24f2df216f392ee74b925e76f91894ba6fa4a8da73d24eb31de993faad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:48:50.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.666258307 +0000 UTC m=+0.341436368 container init 9360e24f2df216f392ee74b925e76f91894ba6fa4a8da73d24eb31de993faad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:48:50.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.669344125 +0000 UTC m=+0.344522182 container start 9360e24f2df216f392ee74b925e76f91894ba6fa4a8da73d24eb31de993faad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:48:50.961 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.741333309 +0000 UTC m=+0.416511376 container attach 9360e24f2df216f392ee74b925e76f91894ba6fa4a8da73d24eb31de993faad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-22T06:48:51.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Apr 22 06:48:50 smithi112 podman[67408]: 2024-04-22 06:48:50.96381916 +0000 UTC m=+0.638997209 container died 9360e24f2df216f392ee74b925e76f91894ba6fa4a8da73d24eb31de993faad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:48:51.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T06:48:51.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:51.586 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T06:48:51.586 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T06:48:51.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.2 2024-04-22T06:48:51.912 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:51 smithi177 systemd[1]: Stopping Ceph osd.2 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:52.239 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2[36481]: 2024-04-22T06:48:51.913+0000 7f0427863700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:52.240 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2[36481]: 2024-04-22T06:48:51.913+0000 7f0427863700 -1 osd.2 117 *** Got signal Terminated *** 2024-04-22T06:48:52.240 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2[36481]: 2024-04-22T06:48:51.913+0000 7f0427863700 -1 osd.2 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:48:57.627 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:57 smithi177 podman[68494]: 2024-04-22 06:48:57.368916303 +0000 UTC m=+5.579168496 container died bec8fdf72983a5b1854ef4b1c7e53b86a86c94247016623808988370ecdb7109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-22T06:48:57.627 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:57 smithi177 podman[68494]: 2024-04-22 06:48:57.62994578 +0000 UTC m=+5.840197957 container cleanup bec8fdf72983a5b1854ef4b1c7e53b86a86c94247016623808988370ecdb7109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-22T06:48:57.627 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:57 smithi177 bash[68494]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2 2024-04-22T06:48:58.171 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:57 smithi177 podman[68510]: 2024-04-22 06:48:57.88099043 +0000 UTC m=+0.513539782 container remove bec8fdf72983a5b1854ef4b1c7e53b86a86c94247016623808988370ecdb7109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T06:48:58.482 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.173639718 +0000 UTC m=+0.094634204 container create 98618719ab015c4e4ad163ae2a89c2c4dd1462d05ffb0321f660aa1ee90b8724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-22T06:48:58.482 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.096936884 +0000 UTC m=+0.017931370 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:48:58.739 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.484628186 +0000 UTC m=+0.405622672 container init 98618719ab015c4e4ad163ae2a89c2c4dd1462d05ffb0321f660aa1ee90b8724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS) 2024-04-22T06:48:58.740 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.48807395 +0000 UTC m=+0.409068439 container start 98618719ab015c4e4ad163ae2a89c2c4dd1462d05ffb0321f660aa1ee90b8724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T06:48:58.740 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.567965549 +0000 UTC m=+0.488960047 container attach 98618719ab015c4e4ad163ae2a89c2c4dd1462d05ffb0321f660aa1ee90b8724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T06:48:59.034 INFO:journalctl@ceph.osd.2.smithi177.stdout:Apr 22 06:48:58 smithi177 podman[68606]: 2024-04-22 06:48:58.78155738 +0000 UTC m=+0.702551860 container died 98618719ab015c4e4ad163ae2a89c2c4dd1462d05ffb0321f660aa1ee90b8724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:48:59.303 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T06:48:59.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:48:59.344 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T06:48:59.344 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T06:48:59.344 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.3 2024-04-22T06:48:59.655 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:48:59 smithi105 systemd[1]: Stopping Ceph osd.3 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:48:59.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:48:59 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:48:59.657+0000 7ff1d43d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:48:59.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:48:59 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:48:59.657+0000 7ff1d43d1700 -1 osd.3 117 *** Got signal Terminated *** 2024-04-22T06:48:59.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:48:59 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:48:59.657+0000 7ff1d43d1700 -1 osd.3 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:02.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:49:02.592+0000 7ff1d09c5700 -1 osd.3 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:36.692939+0000 front 2024-04-22T06:48:36.692920+0000 (oldest deadline 2024-04-22T06:49:01.993092+0000) 2024-04-22T06:49:02.919 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:02 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:02.622+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:03.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:03.404+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:03.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3[49487]: 2024-04-22T06:49:03.630+0000 7ff1d09c5700 -1 osd.3 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:36.692939+0000 front 2024-04-22T06:48:36.692920+0000 (oldest deadline 2024-04-22T06:49:01.993092+0000) 2024-04-22T06:49:03.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:03 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:03.661+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:04.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:04 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:04.014+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:04.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:04.370+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:04.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:04.576+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:04.971 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:04 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:04.693+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:05.239 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:05 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:05.050+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:05.240 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:04 smithi105 podman[91996]: 2024-04-22 06:49:04.973812117 +0000 UTC m=+5.434546190 container died a1352d1bbb3a8550b9adaaa2c79ebfd22e4572bec30243e0558323e5c2e1a3b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:49:05.647 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:05 smithi105 podman[91996]: 2024-04-22 06:49:05.340574658 +0000 UTC m=+5.801308721 container cleanup a1352d1bbb3a8550b9adaaa2c79ebfd22e4572bec30243e0558323e5c2e1a3b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T06:49:05.648 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:05 smithi105 bash[91996]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3 2024-04-22T06:49:05.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:05.363+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:05.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:05.557+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:05.918 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:05 smithi105 podman[92025]: 2024-04-22 06:49:05.650523345 +0000 UTC m=+0.678947410 container remove a1352d1bbb3a8550b9adaaa2c79ebfd22e4572bec30243e0558323e5c2e1a3b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:49:05.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:05 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:05.737+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:05.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:05 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:05.640+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:06.292 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:06 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:06.095+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:06.293 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:05 smithi105 podman[92117]: 2024-04-22 06:49:05.959795968 +0000 UTC m=+0.105624196 container create 417950de349b69da9e35e8837ff6c06c6df14d1f0d88f81eed607cd5a05008ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:06.293 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:05 smithi105 podman[92117]: 2024-04-22 06:49:05.872120446 +0000 UTC m=+0.017948684 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:06.293 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:06 smithi105 podman[92117]: 2024-04-22 06:49:06.195918571 +0000 UTC m=+0.341746814 container init 417950de349b69da9e35e8837ff6c06c6df14d1f0d88f81eed607cd5a05008ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-22T06:49:06.293 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:06 smithi105 podman[92117]: 2024-04-22 06:49:06.199197096 +0000 UTC m=+0.345025324 container start 417950de349b69da9e35e8837ff6c06c6df14d1f0d88f81eed607cd5a05008ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:49:06.293 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:06 smithi105 podman[92117]: 2024-04-22 06:49:06.295390067 +0000 UTC m=+0.441218317 container attach 417950de349b69da9e35e8837ff6c06c6df14d1f0d88f81eed607cd5a05008ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:06.668 INFO:journalctl@ceph.osd.3.smithi105.stdout:Apr 22 06:49:06 smithi105 podman[92117]: 2024-04-22 06:49:06.491093534 +0000 UTC m=+0.636921762 container died 417950de349b69da9e35e8837ff6c06c6df14d1f0d88f81eed607cd5a05008ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-22T06:49:06.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:06.318+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:06.739 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:06.536+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:06.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:06 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:06.675+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:07.063 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:06 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:06.712+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:07.136 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T06:49:07.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:07.178 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T06:49:07.178 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T06:49:07.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.4 2024-04-22T06:49:07.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:07 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:07.065+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:07.458 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:07 smithi112 systemd[1]: Stopping Ceph osd.4 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:07.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:07 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:07.361+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:07.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:07 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:07.564+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:07.917 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:07 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:07.667+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:07.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:07 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:07.719+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:07.958 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:07 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:49:07.496+0000 7f9df9126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:07.958 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:07 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:49:07.496+0000 7f9df9126700 -1 osd.4 117 *** Got signal Terminated *** 2024-04-22T06:49:07.958 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:07 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:49:07.496+0000 7f9df9126700 -1 osd.4 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:08.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:08 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:08.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:08.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:08 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:08.314+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:08.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:08 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:08.538+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:08.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:08 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:08.679+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:09.106 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:08 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:08.709+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:09.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:09.108+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:09.571 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:09.322+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:09.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:09 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:09.702+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:09.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:09 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:09.573+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:10.092 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:09.726+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:10.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:10.094+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:10.617 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:10 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:10.320+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:10.617 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:10 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:10.409+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:10.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:10 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:10.237+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:10.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:10 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:10.619+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:10.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:10 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:10.619+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:11.075 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:10 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:10.718+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:11.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:10 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:10.717+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:11.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:11.077+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:11.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:11.077+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:11.551 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:11 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:11.218+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:11.640 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:11 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:11.432+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:11.640 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:11 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:11.359+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:11.640 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:11 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:11.359+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:11.958 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:11 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4[42656]: 2024-04-22T06:49:11.553+0000 7f9df571a700 -1 osd.4 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.696291+0000 front 2024-04-22T06:48:47.696319+0000 (oldest deadline 2024-04-22T06:49:11.196463+0000) 2024-04-22T06:49:11.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:11 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:11.739+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:11.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:11 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:11.642+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:11.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:11 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:11.642+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:12.080 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:11.672+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:12.080 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:11 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:11.672+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:12.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:12.082+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:12.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:12.082+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:12.648 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:12 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:12.390+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:12.648 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:12 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:12.368+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:12.648 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:12 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:12.368+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:12.688 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:12 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:12.249+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:12.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:12 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:12.690+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:12.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:12 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:12.650+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:12.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:12 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:12.650+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:13.036 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:12.682+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:13.036 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:12 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:12.682+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:13.294 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:12 smithi112 podman[67537]: 2024-04-22 06:49:12.982492223 +0000 UTC m=+5.597606099 container died c12e645b8606d3e56344859ad6254343d993a6e104550a56f35a374156bca848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-22T06:49:13.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:13.038+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:13.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:13.038+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:13.629 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:13 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:13.296+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:13.629 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:13 smithi112 podman[67537]: 2024-04-22 06:49:13.381017228 +0000 UTC m=+5.996131103 container cleanup c12e645b8606d3e56344859ad6254343d993a6e104550a56f35a374156bca848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:49:13.630 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:13 smithi112 bash[67537]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4 2024-04-22T06:49:13.690 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:13 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:13.403+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:13.690 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:13 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:13.386+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:13.690 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:13 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:13.386+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:13.948 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:13 smithi112 podman[67557]: 2024-04-22 06:49:13.632081293 +0000 UTC m=+0.649682690 container remove c12e645b8606d3e56344859ad6254343d993a6e104550a56f35a374156bca848 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:49:13.949 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:13 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:13.713+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:13.949 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:13 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:13.713+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:13.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:13 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:13.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:13.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:13 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:13.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:14.084 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:13.677+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:14.084 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:13 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:13.677+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:14.208 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:13 smithi112 podman[67648]: 2024-04-22 06:49:13.854116766 +0000 UTC m=+0.018231994 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:14.208 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:13 smithi112 podman[67648]: 2024-04-22 06:49:13.957748039 +0000 UTC m=+0.121863267 container create 2ad60bb566a0d6717dc41e224e93d3e3db6bf771d3a11ceb29dafc813e856cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:49:14.209 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:14 smithi112 podman[67648]: 2024-04-22 06:49:14.152010569 +0000 UTC m=+0.316125797 container init 2ad60bb566a0d6717dc41e224e93d3e3db6bf771d3a11ceb29dafc813e856cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:49:14.209 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:14 smithi112 podman[67648]: 2024-04-22 06:49:14.155296071 +0000 UTC m=+0.319411300 container start 2ad60bb566a0d6717dc41e224e93d3e3db6bf771d3a11ceb29dafc813e856cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:14.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:14.086+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:14.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:14.086+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:14.670 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:14.442+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:14.670 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:14.356+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:14.670 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:14.356+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:14.671 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:14.324+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:14.671 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:14 smithi112 podman[67648]: 2024-04-22 06:49:14.243326013 +0000 UTC m=+0.407441254 container attach 2ad60bb566a0d6717dc41e224e93d3e3db6bf771d3a11ceb29dafc813e856cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:49:14.671 INFO:journalctl@ceph.osd.4.smithi112.stdout:Apr 22 06:49:14 smithi112 podman[67648]: 2024-04-22 06:49:14.446828814 +0000 UTC m=+0.610944047 container died 2ad60bb566a0d6717dc41e224e93d3e3db6bf771d3a11ceb29dafc813e856cd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:49:14.917 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:14.632+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:14.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:14.632+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:14.936 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:14.671+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:14.936 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:14 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:14.672+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:14.964 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T06:49:14.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:14.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:14.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:14 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:14.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:15.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:15.004 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T06:49:15.004 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T06:49:15.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.5 2024-04-22T06:49:15.305 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 systemd[1]: Stopping Ceph osd.5 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:15.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:15.108+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:15.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:15.108+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:15.457+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:15.307+0000 7f2826a63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:15.307+0000 7f2826a63700 -1 osd.5 117 *** Got signal Terminated *** 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:15.307+0000 7f2826a63700 -1 osd.5 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:15.324+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:15.652 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:15.324+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:15.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:15 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:15.320+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:15.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:15.636+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:15.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:15 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:15.636+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:15.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:15.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:15.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:15 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:15.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:16.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:15 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:15.710+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:16.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:15 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:15.710+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:16.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:16.145+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:16.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:16.145+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:16.668 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:16 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:16.290+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:16.683 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:16 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:16.477+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:16.683 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:16 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:16.296+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:16.683 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:16 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:16.296+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:16.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:16 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:16.670+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:16.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:16 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:16.670+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:16.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:16 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:16.684+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:16.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:16 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:16.684+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:17.167 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:16.674+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:17.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:16 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:16.674+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:17.659 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:17.180+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:17.659 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:17.180+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:17.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:17 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:17.254+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:17.732 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:17 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:17.514+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:17.732 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:17 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:17.322+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:17.733 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:17 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:17.322+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:17.917 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:17.661+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:17.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:17 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:17.661+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:17.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:17 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:17.734+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:17.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:17 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:17.734+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:18.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:17 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:17.715+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:18.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:17 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:17.715+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:18.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:18 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:18.150+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:18.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:18 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:18.150+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:18.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:18 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:18.255+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:18.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:18.354+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:18.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:18.354+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:18.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:18.537+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:19.118 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:18 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:18.706+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:19.118 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:18 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:18.706+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:19.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:18 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:18.753+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:19.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:18 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:18.753+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:19.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:18.780+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:19.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:18 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:18.780+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:19.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:19.120+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:19.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:19.120+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:19.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:19 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:19.236+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:19.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:19 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:19.236+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:19.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:19.533+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:19.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:19.354+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.955450+0000 front 2024-04-22T06:48:40.955318+0000 (oldest deadline 2024-04-22T06:49:02.654836+0000) 2024-04-22T06:49:19.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:19.354+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.156722+0000 front 2024-04-22T06:48:46.156979+0000 (oldest deadline 2024-04-22T06:49:10.856114+0000) 2024-04-22T06:49:19.739 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5[42973]: 2024-04-22T06:49:19.355+0000 7f281f04f700 -1 osd.5 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:54.356818+0000 front 2024-04-22T06:48:54.356861+0000 (oldest deadline 2024-04-22T06:49:19.056990+0000) 2024-04-22T06:49:20.100 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:19.737+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:20.100 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:19.737+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:20.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:19 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:19.714+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:20.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:19 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:19.715+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:20.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:19 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:19.715+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:20.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:19.779+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:20.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:19 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:19.779+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:20.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:20 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:20.102+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:20.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:20 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:20.102+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:20.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:20 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:20.102+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:20.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:20.245+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:20.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:20.245+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:20.879 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:20 smithi177 podman[68727]: 2024-04-22 06:49:20.629472611 +0000 UTC m=+5.422678193 container died 317037bfe4f8f17fb285512233315cecd1af866fde47dffb4bfe22a87f810393 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416) 2024-04-22T06:49:20.880 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:20 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:20.576+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:20.880 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:20 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:20.789+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:20.880 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:20 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:20.789+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:21.130 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:20 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:20.748+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:21.130 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:20 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:20.748+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:21.147 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:20 smithi177 podman[68727]: 2024-04-22 06:49:20.88236008 +0000 UTC m=+5.675565670 container cleanup 317037bfe4f8f17fb285512233315cecd1af866fde47dffb4bfe22a87f810393 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-22T06:49:21.147 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:20 smithi177 bash[68727]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5 2024-04-22T06:49:21.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:20.712+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:21.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:20.712+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:21.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:20 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:20.712+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:21.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:21.131+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:21.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:21.131+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:21.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:21.131+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:21.489 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68744]: 2024-04-22 06:49:21.150321765 +0000 UTC m=+0.519431204 container remove 317037bfe4f8f17fb285512233315cecd1af866fde47dffb4bfe22a87f810393 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True) 2024-04-22T06:49:21.691 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:21 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:21.212+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:21.691 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:21 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:21.212+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:21.565+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:21.565+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68837]: 2024-04-22 06:49:21.401021051 +0000 UTC m=+0.018294655 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68837]: 2024-04-22 06:49:21.509492704 +0000 UTC m=+0.126766341 container create 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68837]: 2024-04-22 06:49:21.720163165 +0000 UTC m=+0.337436759 container init 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux ) 2024-04-22T06:49:21.794 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68837]: 2024-04-22 06:49:21.723218269 +0000 UTC m=+0.340491862 container start 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-22T06:49:21.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:21 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:21.693+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:21.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:21 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:21.693+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:21.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:21 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:21.693+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:22.088 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:21.776+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:22.088 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:21 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:21.776+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:22.239 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:21 smithi177 podman[68837]: 2024-04-22 06:49:21.795162255 +0000 UTC m=+0.412435856 container attach 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-22T06:49:22.239 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:22 smithi177 podman[68837]: 2024-04-22 06:49:22.014153011 +0000 UTC m=+0.631426602 container died 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:49:22.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:21.825+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:22.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:21.825+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:22.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:21 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:21.825+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:22.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:22.090+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:22.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:22.090+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:22.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:22.090+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:22.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T06:49:22.592 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:22.541+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:22.592 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:22.541+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:22.592 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:22 smithi177 podman[68863]: 2024-04-22 06:49:22.548930799 +0000 UTC m=+0.529833219 container remove 3e2754c4145b7f828c85f5dd0fc831626578b3923a0e3e6129f71186929e4e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:49:22.592 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:22 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-22T06:49:22.593 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:22 smithi177 systemd[1]: Stopped Ceph osd.5 for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:49:22.593 INFO:journalctl@ceph.osd.5.smithi177.stdout:Apr 22 06:49:22 smithi177 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.5.service: Consumed 15.946s CPU time. 2024-04-22T06:49:22.607 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:22.607 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T06:49:22.607 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T06:49:22.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.6 2024-04-22T06:49:22.667 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:22 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:22.246+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:22.667 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:22 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:22.246+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:22.899 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 systemd[1]: Stopping Ceph osd.6 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:22.899 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.741+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:22.899 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.741+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:22.899 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.741+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:22.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:22 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:22.669+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:22.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:22 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:22.669+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:22.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:22 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:22.669+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:22.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:22.806+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:22.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:22.806+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:22.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:22 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:22.806+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:23.167 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:23.051+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:23.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:23.051+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:23.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:23.051+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:23.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.900+0000 7fdfecd62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:23.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.900+0000 7fdfecd62700 -1 osd.6 117 *** Got signal Terminated *** 2024-04-22T06:49:23.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:22.900+0000 7fdfecd62700 -1 osd.6 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:23.693 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:23 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:23.271+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:23.693 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:23 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:23.271+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:23.814 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:23 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:23.502+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:23.814 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:23 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:23.502+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:23.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:23 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:23.695+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:23.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:23 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:23.695+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:23.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:23 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:23.695+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:24.167 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:23.766+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:24.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:23.766+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:24.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:23 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:23.766+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:24.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:24.012+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:24.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:24.012+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:24.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:24.012+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:24.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:23 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:23.816+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:24.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:23 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:23.816+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:24.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:23 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:23.816+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:24.678 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:24.281+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:24.678 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:24.281+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:24.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:24.457+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:24.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:24.457+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:24.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:24.680+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:24.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:24.680+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:24.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:24 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:24.680+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:25.167 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:24.803+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:25.167 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:24.803+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:25.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:24 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:24.803+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:25.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.039+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:25.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.039+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:25.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.039+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:25.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:24.787+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:25.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:24.787+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:25.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:24 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:24.787+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:25.686 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:25.273+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:25.687 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:25.273+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:25.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:25.443+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:25.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:25.443+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:25.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:25.689+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:25.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:25.689+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:25.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:25 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:25.689+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:25.820+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:25.820+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:25.820+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:25.820+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.844109+0000 front 2024-04-22T06:49:02.844141+0000 (oldest deadline 2024-04-22T06:49:25.744150+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:26.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:25 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:25.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:26.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:25.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:26.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:25.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:26.239 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:25 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:25.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:26.698 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:26.255+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:26.699 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:26.255+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:26.714 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:26.405+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:26.714 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:26.405+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:26.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:26.701+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:26.959 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:26.701+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:26.959 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:26 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:26.701+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:26.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:26.716+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:26.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:26.716+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:26.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:26.716+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:26.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:26 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:26.716+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:26 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:26.843+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:26 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:26.843+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:26 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:26.843+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:26 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:26.843+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.844109+0000 front 2024-04-22T06:49:02.844141+0000 (oldest deadline 2024-04-22T06:49:25.744150+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:27.034+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:27.034+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:27.034+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:27.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:27.034+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:27.676 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:27.373+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:27.676 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:27.373+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:27.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:27.232+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:27.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:27.233+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:27.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:27.233+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:27.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:27.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:27.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:27.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:27.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:27.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:27.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:27 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:27.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:27.831+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.040984+0000 front 2024-04-22T06:48:37.041034+0000 (oldest deadline 2024-04-22T06:49:02.341113+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:27.831+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.242030+0000 front 2024-04-22T06:48:48.241988+0000 (oldest deadline 2024-04-22T06:49:11.142097+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:27.831+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:51.142707+0000 front 2024-04-22T06:48:51.142736+0000 (oldest deadline 2024-04-22T06:49:22.343340+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:27 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6[56494]: 2024-04-22T06:49:27.831+0000 7fdfe534e700 -1 osd.6 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.844109+0000 front 2024-04-22T06:49:02.844141+0000 (oldest deadline 2024-04-22T06:49:25.744150+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.003+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.003+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.003+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:28.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.003+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:28.203 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:27.728+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:28.203 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:27.728+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:28.203 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:27.728+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:28.458 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:28.205+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:28.458 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:28.205+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:28.458 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:28.205+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:28.629 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:28.364+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:28.629 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:28.364+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:28.716 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:28 smithi105 podman[92224]: 2024-04-22 06:49:28.44091063 +0000 UTC m=+5.637182725 container died a55b21bfd020a54fb2267844afc68e8ded2763a0e1a1bb81857a4a07b0dd75a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:28.988 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:28 smithi105 podman[92224]: 2024-04-22 06:49:28.817944311 +0000 UTC m=+6.014216402 container cleanup a55b21bfd020a54fb2267844afc68e8ded2763a0e1a1bb81857a4a07b0dd75a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD) 2024-04-22T06:49:28.988 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:28 smithi105 bash[92224]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6 2024-04-22T06:49:28.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:28.631+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:28.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:28.631+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:28.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:28.631+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:28.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:28 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:28.631+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:29.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:28.778+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:29.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:28.778+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:29.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:28 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:28.778+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:29.357 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92241]: 2024-04-22 06:49:29.069209203 +0000 UTC m=+0.630781154 container remove a55b21bfd020a54fb2267844afc68e8ded2763a0e1a1bb81857a4a07b0dd75a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-22T06:49:29.357 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.989+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:29.357 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.989+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:29.357 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.989+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:29.357 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:28 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:28.989+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:29.661 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.262989674 +0000 UTC m=+0.018040605 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:29.662 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.361586056 +0000 UTC m=+0.116636972 container create d327a062cfe02a7738c1cc6da46b5f154d66c2d5b1e33c2622e27cf32faad474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-22T06:49:29.662 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.589462715 +0000 UTC m=+0.344513631 container init d327a062cfe02a7738c1cc6da46b5f154d66c2d5b1e33c2622e27cf32faad474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-22T06:49:29.662 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.592584745 +0000 UTC m=+0.347635666 container start d327a062cfe02a7738c1cc6da46b5f154d66c2d5b1e33c2622e27cf32faad474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:49:29.662 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.66414916 +0000 UTC m=+0.419200097 container attach d327a062cfe02a7738c1cc6da46b5f154d66c2d5b1e33c2622e27cf32faad474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T06:49:29.676 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:29.389+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:29.676 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:29.389+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:29.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:29.235+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:29.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:29.235+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:29.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:29.235+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:29.918 INFO:journalctl@ceph.osd.6.smithi105.stdout:Apr 22 06:49:29 smithi105 podman[92331]: 2024-04-22 06:49:29.88691732 +0000 UTC m=+0.641968228 container died d327a062cfe02a7738c1cc6da46b5f154d66c2d5b1e33c2622e27cf32faad474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:29.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:29.677+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:29.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:29.677+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:29.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:29.677+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:29.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:29 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:29.677+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:30.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:29.779+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:30.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:29.779+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:30.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:29.779+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:30.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:29 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:29.779+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:30.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.005+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:30.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.005+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:30.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.005+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:30.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.005+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:30.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T06:49:30.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:30.528 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T06:49:30.528 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T06:49:30.529 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.7 2024-04-22T06:49:30.554 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.250+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:30.554 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.250+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:30.554 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.250+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:30.669 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:30.376+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:30.669 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:30.376+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:30.669 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:30.376+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:30.830 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 systemd[1]: Stopping Ceph osd.7 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:30.830 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:30.804+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:30.831 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:30.804+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:30.831 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:30.804+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:30.831 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:30.804+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:30.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:30.670+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:30.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:30.670+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:30.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:30.670+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:30.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:30 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:30.670+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:31.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.977+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:31.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.977+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:31.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.977+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:31.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:30 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:30.977+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:31.208 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.833+0000 7fb1714f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:31.208 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.833+0000 7fb1714f8700 -1 osd.7 117 *** Got signal Terminated *** 2024-04-22T06:49:31.208 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:30.833+0000 7fb1714f8700 -1 osd.7 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:31.663 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:31.341+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:31.663 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:31.341+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:31.663 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:31.341+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:31.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:31.258+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:31.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:31.258+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:31.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:31.258+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:31.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:31.664+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:31.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:31.664+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:31.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:31.664+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:31.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:31 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:31.664+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:32.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:31.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:32.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:31.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:32.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:31.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:32.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:31 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:31.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:32.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:32.024+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:32.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:32.024+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:32.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:32.024+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:32.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:32 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:32.024+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:32.481 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:32.264+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:32.481 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:32.264+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:32.481 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:32.264+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:32.714 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:32.390+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:32.714 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:32.390+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:32.714 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:32.390+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:32.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:32.713+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:32.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:32.713+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:32.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:32.713+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:32.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:32 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:32.713+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:33.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:32.805+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:33.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:32.805+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:33.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:32.805+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:33.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:32 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:32.805+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:33.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:33.058+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:33.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:33.058+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:33.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:33.058+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:33.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:33 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:33.058+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:33.678 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:33.418+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:33.678 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:33.418+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:33.679 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:33.418+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:33.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:33.229+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:33.709 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:33.229+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:33.709 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:33.229+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:33.951 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:33.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:33.951 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:33.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:33.951 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:33.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:33.951 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:33 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:33.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:34.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:33.849+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:34.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:33.849+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:34.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:33.849+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:34.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:33.849+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:34.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:33 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:33.849+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:34.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:34.091+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:34.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:34.091+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:34.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:34.091+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:34.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:34.091+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:34.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:34 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:34.091+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:34.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:34.277+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:34.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:34.277+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:34.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:34.277+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:34.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:34.448+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:34.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:34.448+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:34.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:34.448+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:34.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:34.660+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:34.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:34.660+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:34.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:34.660+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:34.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:34 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:34.660+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:35.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:34.833+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:35.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:34.833+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:35.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:34.833+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:35.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:34.833+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:35.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:34.833+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:35.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:35.054+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:35.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:35.054+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:35.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:35.054+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:35.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:35.054+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:35.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:35 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:35.054+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:35.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:35.293+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.991434+0000 front 2024-04-22T06:48:48.991510+0000 (oldest deadline 2024-04-22T06:49:10.091594+0000) 2024-04-22T06:49:35.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:35.293+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.592923+0000 front 2024-04-22T06:48:53.592637+0000 (oldest deadline 2024-04-22T06:49:18.892661+0000) 2024-04-22T06:49:35.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7[49192]: 2024-04-22T06:49:35.293+0000 7fb16daec700 -1 osd.7 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:58.893315+0000 front 2024-04-22T06:48:58.893378+0000 (oldest deadline 2024-04-22T06:49:26.494077+0000) 2024-04-22T06:49:35.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:35.459+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:35.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:35.459+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:35.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:35.459+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:35.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:35.626+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:35.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:35.626+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:35.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:35.626+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:35.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:35 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:35.626+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:36.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:35.796+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:36.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:35.796+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:36.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:35.796+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:36.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:35.796+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:36.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:35 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:35.796+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:36.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:36.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:36.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:36.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:36.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:36.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:36.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:36.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:36.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:36 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:36.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:36.571 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:36 smithi112 podman[67754]: 2024-04-22 06:49:36.316901048 +0000 UTC m=+5.589517305 container died 0560ac1df4e49cc6d752d09f6ff98a3dc269395c7c25f5af44fdf42eb4ab5554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-22T06:49:36.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:36.453+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:36.453+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:36.453+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:36.453+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:36.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:36.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:36.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:36.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:36.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:36 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:36.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:36 smithi112 podman[67754]: 2024-04-22 06:49:36.574207945 +0000 UTC m=+5.846824244 container cleanup 0560ac1df4e49cc6d752d09f6ff98a3dc269395c7c25f5af44fdf42eb4ab5554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:36 smithi112 bash[67754]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:36 smithi112 podman[67784]: 2024-04-22 06:49:36.750518091 +0000 UTC m=+0.429256024 container remove 0560ac1df4e49cc6d752d09f6ff98a3dc269395c7c25f5af44fdf42eb4ab5554 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:36.816+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:36.816+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:36.816+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:36.958 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:36.816+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:36.959 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:36 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:36.816+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:37.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:37.057+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:37.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:37.057+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:37.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:37.057+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:37.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:37.057+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:37.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:37 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:37.057+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:37.435 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.151469507 +0000 UTC m=+0.096306318 container create def0619190938d544409d02be35a73279a6854c4d6e8944396d65259d74edb46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-22T06:49:37.435 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.072616886 +0000 UTC m=+0.017453697 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:37.435 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.362088304 +0000 UTC m=+0.306925115 container init def0619190938d544409d02be35a73279a6854c4d6e8944396d65259d74edb46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-22T06:49:37.435 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.365112995 +0000 UTC m=+0.309949812 container start def0619190938d544409d02be35a73279a6854c4d6e8944396d65259d74edb46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-22T06:49:37.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.437022257 +0000 UTC m=+0.381859076 container attach def0619190938d544409d02be35a73279a6854c4d6e8944396d65259d74edb46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:49:37.708 INFO:journalctl@ceph.osd.7.smithi112.stdout:Apr 22 06:49:37 smithi112 podman[67872]: 2024-04-22 06:49:37.655641293 +0000 UTC m=+0.600478096 container died def0619190938d544409d02be35a73279a6854c4d6e8944396d65259d74edb46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-22T06:49:37.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:37.464+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:37.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:37.464+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:37.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:37.464+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:37.739 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:37.464+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:37.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:37.609+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:37.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:37.609+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:37.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:37.609+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:37.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:37.609+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:37.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:37 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:37.609+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:38.188 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:37 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:37.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:38.188 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:37 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:37.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:38.188 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:37 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:37.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:38.188 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:37 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:37.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:38.188 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:37 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:37.819+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:38.205 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T06:49:38.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:38.282 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T06:49:38.282 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-22T06:49:38.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.8 2024-04-22T06:49:38.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:38.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:38.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:38.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:38.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:38.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:38.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:38.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:38.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:38 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:38.101+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 systemd[1]: Stopping Ceph osd.8 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.447+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.447+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.447+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.447+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.608+0000 7ff0f79d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.608+0000 7ff0f79d2700 -1 osd.8 117 *** Got signal Terminated *** 2024-04-22T06:49:38.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:38.608+0000 7ff0f79d2700 -1 osd.8 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:38.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:38.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:38.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:38.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:38.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:38.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:38.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:38.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:38.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:38 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:38.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:39.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:38.828+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:39.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:38.828+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:39.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:38.828+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:39.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:38.828+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:39.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:38.828+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:39.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:39.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:39.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:39.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:39.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:39.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:39.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:39.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:39.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:39 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:39.070+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:39.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:39.633+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:39.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:39.633+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:39.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:39.633+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:39.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:39.633+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:39.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:39.633+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:39.741 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:39.470+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:39.741 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:39.470+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:39.741 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:39.470+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:39.741 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:39 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:39.470+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:40.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:39 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:39.846+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:40.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:39 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:39.846+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:40.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:39 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:39.846+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:40.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:39 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:39.846+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:40.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:39 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:39.846+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:40.417 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.026+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:40.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.026+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:40.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.026+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:40.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.026+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:40.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.026+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:40.665+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:40.665+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:40.665+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:40.665+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:40.665+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:40.449+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:40.449+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:40.449+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:40.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:40 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:40.449+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:41.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:40 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:40.866+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:41.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:40 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:40.866+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:41.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:40 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:40.866+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:41.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:40 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:40.866+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:41.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:40 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:40.866+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:41.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:41.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:41.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:41.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:41.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:40 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:40.990+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:41.699+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:41.490+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:41.490+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:41.490+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:41.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:41 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:41.490+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:42.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:41.966+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:42.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:41.966+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:42.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:41.966+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:42.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:41.966+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:42.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:41 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:41.966+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:42.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:41 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:41.845+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:42.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:41 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:41.845+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:42.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:41 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:41.845+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:42.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:41 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:41.845+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:42.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:41 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:41.845+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:42.683+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:42.990 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:42.499+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:42.991 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:42.499+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:42.991 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:42.499+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:42.991 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:42.499+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:43.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:42.918+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:43.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:42.918+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:43.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:42.918+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:43.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:42.918+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:43.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:42 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:42.918+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:43.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:42 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:42.894+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:43.739 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:43.669+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:43.488+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:46.745976+0000 front 2024-04-22T06:48:46.745467+0000 (oldest deadline 2024-04-22T06:49:10.245447+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:43.488+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.647089+0000 front 2024-04-22T06:48:56.647192+0000 (oldest deadline 2024-04-22T06:49:21.347268+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:43.488+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:04.249263+0000 front 2024-04-22T06:49:04.249127+0000 (oldest deadline 2024-04-22T06:49:29.548171+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:43.488+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:12.449801+0000 front 2024-04-22T06:49:12.449237+0000 (oldest deadline 2024-04-22T06:49:35.949259+0000) 2024-04-22T06:49:43.740 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:43 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8[49976]: 2024-04-22T06:49:43.488+0000 7ff0effbe700 -1 osd.8 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.350397+0000 front 2024-04-22T06:49:19.350349+0000 (oldest deadline 2024-04-22T06:49:42.850494+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:44.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:43 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:43.882+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:44.279 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 podman[68954]: 2024-04-22 06:49:44.014734335 +0000 UTC m=+5.533181819 container died 0107623d05b8c7fb6b210ceb715132b1c3fc8862bb90f14c41ab10c9939e614c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:44.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:43.950+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:44.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:43.950+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:44.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:43.950+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:44.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:43.950+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:44.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:43 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:43.950+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:44.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 podman[68954]: 2024-04-22 06:49:44.375319881 +0000 UTC m=+5.893767369 container cleanup 0107623d05b8c7fb6b210ceb715132b1c3fc8862bb90f14c41ab10c9939e614c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-22T06:49:44.607 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 bash[68954]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8 2024-04-22T06:49:44.925 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 podman[68971]: 2024-04-22 06:49:44.610038849 +0000 UTC m=+0.593200475 container remove 0107623d05b8c7fb6b210ceb715132b1c3fc8862bb90f14c41ab10c9939e614c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-22T06:49:44.925 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:44.925 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:44.925 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:44.925 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:44.926 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:44.926 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:44 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:44.638+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:45.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:44.900+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:45.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:44.900+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:45.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:44.900+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:45.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:44.900+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:45.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:44 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:44.900+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:45.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:44 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:44.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:45.211 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 podman[69060]: 2024-04-22 06:49:44.927564309 +0000 UTC m=+0.110157861 container create 52eed92ab8645d5e0ad90df6ce0ebef425905a7889d6eab0889144eb5f518351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-22T06:49:45.211 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:44 smithi177 podman[69060]: 2024-04-22 06:49:44.83529404 +0000 UTC m=+0.017887617 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:45.211 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:45 smithi177 podman[69060]: 2024-04-22 06:49:45.139196879 +0000 UTC m=+0.321790428 container init 52eed92ab8645d5e0ad90df6ce0ebef425905a7889d6eab0889144eb5f518351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:49:45.211 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:45 smithi177 podman[69060]: 2024-04-22 06:49:45.142387526 +0000 UTC m=+0.324981078 container start 52eed92ab8645d5e0ad90df6ce0ebef425905a7889d6eab0889144eb5f518351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-22T06:49:45.211 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:45 smithi177 podman[69060]: 2024-04-22 06:49:45.213943142 +0000 UTC m=+0.396536692 container attach 52eed92ab8645d5e0ad90df6ce0ebef425905a7889d6eab0889144eb5f518351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:49:45.489 INFO:journalctl@ceph.osd.8.smithi177.stdout:Apr 22 06:49:45 smithi177 podman[69060]: 2024-04-22 06:49:45.433146519 +0000 UTC m=+0.615740069 container died 52eed92ab8645d5e0ad90df6ce0ebef425905a7889d6eab0889144eb5f518351 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:45.956 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:45 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:45.672+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:45.981 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.8.service' 2024-04-22T06:49:46.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:46.022 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-22T06:49:46.022 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-22T06:49:46.022 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.9 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:46.210 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:45 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:45.881+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:45 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:45.923+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 systemd[1]: Stopping Ceph osd.9 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:46.318 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.319+0000 7fa976e6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:46.319 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.319+0000 7fa976e6b700 -1 osd.9 117 *** Got signal Terminated *** 2024-04-22T06:49:46.319 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.319+0000 7fa976e6b700 -1 osd.9 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:46.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:46.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:47.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:46 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:46.885+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:47.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:46 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:46.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:47.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:47 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:47.658+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:48.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:48.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:48.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:48.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:48.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:48.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:47 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:47.899+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:48.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:47 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:47.968+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:48.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:48 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:48.645+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:49.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:49.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:49.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:49.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:49.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:49.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:48 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:48.929+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:49.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:48 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:48.963+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:49.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:49.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:49.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:49.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:49.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:49.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:49 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:49.655+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:50.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:50.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:50.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:50.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:50.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:50.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:49 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:49.947+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:50.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:50 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:50.011+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:27.501323+0000 front 2024-04-22T06:49:27.501341+0000 (oldest deadline 2024-04-22T06:49:49.201320+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:50.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:50.671+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:37.894110+0000 front 2024-04-22T06:48:37.894092+0000 (oldest deadline 2024-04-22T06:49:03.194013+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:48.495001+0000 front 2024-04-22T06:48:48.494984+0000 (oldest deadline 2024-04-22T06:49:10.195035+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:53.597330+0000 front 2024-04-22T06:48:53.597280+0000 (oldest deadline 2024-04-22T06:49:19.496826+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:48:59.497181+0000 front 2024-04-22T06:48:59.497264+0000 (oldest deadline 2024-04-22T06:49:25.397238+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.298615+0000 front 2024-04-22T06:49:08.298575+0000 (oldest deadline 2024-04-22T06:49:33.598510+0000) 2024-04-22T06:49:51.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:17.600369+0000 front 2024-04-22T06:49:17.600273+0000 (oldest deadline 2024-04-22T06:49:45.200527+0000) 2024-04-22T06:49:51.419 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9[63514]: 2024-04-22T06:49:51.052+0000 7fa96f457700 -1 osd.9 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:27.501323+0000 front 2024-04-22T06:49:27.501341+0000 (oldest deadline 2024-04-22T06:49:49.201320+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:51.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:50 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:50.979+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:51.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:51.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:51.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:51.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:51.991 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:51.991 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:51 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:51.647+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:52.159 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:51 smithi105 podman[92449]: 2024-04-22 06:49:51.868042651 +0000 UTC m=+5.651947761 container died 4cb18d0dc259855ef474355fd21b5c41e8d1166d57ab007edee488b961deacf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-22T06:49:52.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:52 smithi105 podman[92449]: 2024-04-22 06:49:52.261747823 +0000 UTC m=+6.045652927 container cleanup 4cb18d0dc259855ef474355fd21b5c41e8d1166d57ab007edee488b961deacf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-22T06:49:52.418 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:52 smithi105 bash[92449]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9 2024-04-22T06:49:52.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:52.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:52.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:52.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:52.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:52.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:52.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:51 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:51.994+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:52.876 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:52 smithi105 podman[92466]: 2024-04-22 06:49:52.521074397 +0000 UTC m=+0.644057893 container remove 4cb18d0dc259855ef474355fd21b5c41e8d1166d57ab007edee488b961deacf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-22T06:49:52.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:52.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:52.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:52.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:52.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:52.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:52 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:52.694+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:53.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:52 smithi105 podman[92554]: 2024-04-22 06:49:52.781704811 +0000 UTC m=+0.017774986 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:49:53.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:52 smithi105 podman[92554]: 2024-04-22 06:49:52.880092407 +0000 UTC m=+0.116162595 container create 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:49:53.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:53 smithi105 podman[92554]: 2024-04-22 06:49:53.098932373 +0000 UTC m=+0.335002528 container init 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:53.168 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:53 smithi105 podman[92554]: 2024-04-22 06:49:53.101944028 +0000 UTC m=+0.338014181 container start 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-22T06:49:53.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:53.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:53.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:53.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:53.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:53.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:53.459 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:52 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:52.963+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:53.668 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:53 smithi105 podman[92554]: 2024-04-22 06:49:53.207627736 +0000 UTC m=+0.443697900 container attach 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:49:53.668 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:53 smithi105 podman[92554]: 2024-04-22 06:49:53.386751207 +0000 UTC m=+0.622821347 container died 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-22T06:49:53.927 INFO:journalctl@ceph.osd.9.smithi105.stdout:Apr 22 06:49:53 smithi105 podman[92580]: 2024-04-22 06:49:53.919302002 +0000 UTC m=+0.521510074 container remove 4e594d30dd389e95e552961975afefe72c0a54cfe3dd7507d87ad70c0fb8fc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:49:53.932 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.9.service' 2024-04-22T06:49:53.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:49:53.970 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-22T06:49:53.970 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-22T06:49:53.971 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.10 2024-04-22T06:49:53.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:53.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:53.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:53.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:53.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:53.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:53 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:53.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:54.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:53 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:53.933+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:54.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 systemd[1]: Stopping Ceph osd.10 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:49:54.708 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.292+0000 7f0554d92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:49:54.708 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.292+0000 7f0554d92700 -1 osd.10 117 *** Got signal Terminated *** 2024-04-22T06:49:54.708 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.292+0000 7f0554d92700 -1 osd.10 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:54.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:54.692+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:55.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:55.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:55.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:55.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:55.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:55.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:55.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:54 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:54.889+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:55.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:55.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:55.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:55.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:55.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:55.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:55 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:55.690+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:56.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:55 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:55.871+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:56.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:56.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:56.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:56.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:56.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:56.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:56 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:56.654+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:57.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:57.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:56 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:56.857+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:57.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:57.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:57.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:57.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:57.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:57.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:57.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:57 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:57.693+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:49:58.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:58.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:58.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:58.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:58.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:58.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:58.209 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:57 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:57.832+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:58.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:58 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:58.663+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:40.024074+0000 front 2024-04-22T06:48:40.024022+0000 (oldest deadline 2024-04-22T06:49:05.323998+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:47.024799+0000 front 2024-04-22T06:48:47.024890+0000 (oldest deadline 2024-04-22T06:49:12.924898+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:52.925994+0000 front 2024-04-22T06:48:52.926072+0000 (oldest deadline 2024-04-22T06:49:18.825253+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:02.926413+0000 front 2024-04-22T06:49:02.926312+0000 (oldest deadline 2024-04-22T06:49:28.826101+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:08.826418+0000 front 2024-04-22T06:49:08.826438+0000 (oldest deadline 2024-04-22T06:49:33.526566+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:19.328240+0000 front 2024-04-22T06:49:19.328099+0000 (oldest deadline 2024-04-22T06:49:42.227832+0000) 2024-04-22T06:49:59.208 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:58 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10[55803]: 2024-04-22T06:49:58.835+0000 7f054d37e700 -1 osd.10 117 heartbeat_check: no reply from 172.21.15.105:6822 osd.6 since back 2024-04-22T06:49:22.228331+0000 front 2024-04-22T06:49:22.228487+0000 (oldest deadline 2024-04-22T06:49:50.428729+0000) 2024-04-22T06:49:59.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:49:59.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:49:59 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:49:59.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:00.008 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:49:59 smithi112 podman[67977]: 2024-04-22 06:49:59.747574806 +0000 UTC m=+5.574095250 container died 0951a4e21c935af1a44da80b08f0aa289f0215643acce76a576b03222806b5ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T06:50:00.375 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[67977]: 2024-04-22 06:50:00.109708554 +0000 UTC m=+5.936228985 container cleanup 0951a4e21c935af1a44da80b08f0aa289f0215643acce76a576b03222806b5ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:50:00.375 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 bash[67977]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10 2024-04-22T06:50:00.708 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[67994]: 2024-04-22 06:50:00.377473311 +0000 UTC m=+0.625843841 container remove 0951a4e21c935af1a44da80b08f0aa289f0215643acce76a576b03222806b5ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587) 2024-04-22T06:50:00.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:00.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:00.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:00.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:00.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:00.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:00.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:00 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:00.678+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:01.020 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[68087]: 2024-04-22 06:50:00.635273587 +0000 UTC m=+0.018205292 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:50:01.020 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[68087]: 2024-04-22 06:50:00.744768363 +0000 UTC m=+0.127700044 container create 29c7dbbef96c28d167fdddd78ced85d4b4d0522176f45a16f2e2e467aa2456c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:01.020 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[68087]: 2024-04-22 06:50:00.95612446 +0000 UTC m=+0.339056156 container init 29c7dbbef96c28d167fdddd78ced85d4b4d0522176f45a16f2e2e467aa2456c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:01.020 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:00 smithi112 podman[68087]: 2024-04-22 06:50:00.95942663 +0000 UTC m=+0.342358320 container start 29c7dbbef96c28d167fdddd78ced85d4b4d0522176f45a16f2e2e467aa2456c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-22T06:50:01.020 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:01 smithi112 podman[68087]: 2024-04-22 06:50:01.022214765 +0000 UTC m=+0.405146450 container attach 29c7dbbef96c28d167fdddd78ced85d4b4d0522176f45a16f2e2e467aa2456c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-22T06:50:01.458 INFO:journalctl@ceph.osd.10.smithi112.stdout:Apr 22 06:50:01 smithi112 podman[68087]: 2024-04-22 06:50:01.247253504 +0000 UTC m=+0.630185182 container died 29c7dbbef96c28d167fdddd78ced85d4b4d0522176f45a16f2e2e467aa2456c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-22T06:50:01.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.10.service' 2024-04-22T06:50:01.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:01.732 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-22T06:50:01.732 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-22T06:50:01.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.11 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:01.680+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:01.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:01 smithi177 systemd[1]: Stopping Ceph osd.11 for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:02.489 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.018+0000 7f80a306b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:02.490 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.018+0000 7f80a306b700 -1 osd.11 117 *** Got signal Terminated *** 2024-04-22T06:50:02.490 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.018+0000 7f80a306b700 -1 osd.11 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T06:50:02.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:02.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:02 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:02.691+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:03.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:03 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:03.667+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:04.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:04.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:04 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:04.714+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:05.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:05.990 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:05 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:05.718+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-04-22T06:48:39.260559+0000 front 2024-04-22T06:48:39.260502+0000 (oldest deadline 2024-04-22T06:49:03.960358+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6804 osd.1 since back 2024-04-22T06:48:43.960891+0000 front 2024-04-22T06:48:43.960786+0000 (oldest deadline 2024-04-22T06:49:09.860908+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6804 osd.2 since back 2024-04-22T06:48:56.862409+0000 front 2024-04-22T06:48:56.862446+0000 (oldest deadline 2024-04-22T06:49:21.562618+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.105:6814 osd.3 since back 2024-04-22T06:49:01.564222+0000 front 2024-04-22T06:49:01.564074+0000 (oldest deadline 2024-04-22T06:49:26.263195+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6812 osd.4 since back 2024-04-22T06:49:09.764567+0000 front 2024-04-22T06:49:09.764657+0000 (oldest deadline 2024-04-22T06:49:36.164685+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.177:6812 osd.5 since back 2024-04-22T06:49:16.165116+0000 front 2024-04-22T06:49:16.165139+0000 (oldest deadline 2024-04-22T06:49:41.465215+0000) 2024-04-22T06:50:07.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:06 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11[56723]: 2024-04-22T06:50:06.744+0000 7f809f65f700 -1 osd.11 117 heartbeat_check: no reply from 172.21.15.112:6820 osd.7 since back 2024-04-22T06:49:31.366943+0000 front 2024-04-22T06:49:31.366869+0000 (oldest deadline 2024-04-22T06:49:57.266749+0000) 2024-04-22T06:50:07.739 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:07 smithi177 podman[69165]: 2024-04-22 06:50:07.442928096 +0000 UTC m=+5.516273763 container died b3c50409993e25b8d45da9b000d3274da4cc96137e4e2281ac5318fd524d4d6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:50:08.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:07 smithi177 podman[69165]: 2024-04-22 06:50:07.751314769 +0000 UTC m=+5.824660436 container cleanup b3c50409993e25b8d45da9b000d3274da4cc96137e4e2281ac5318fd524d4d6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-22T06:50:08.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:07 smithi177 bash[69165]: ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11 2024-04-22T06:50:08.240 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:07 smithi177 podman[69183]: 2024-04-22 06:50:07.960808276 +0000 UTC m=+0.511825707 container remove b3c50409993e25b8d45da9b000d3274da4cc96137e4e2281ac5318fd524d4d6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-22T06:50:08.626 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.188383161 +0000 UTC m=+0.017526045 image pull 1b5e03f260bdcb5b725274fde291494c3b969a1a9de8edaab3dd8caea076bb61 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2 2024-04-22T06:50:08.626 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.286780357 +0000 UTC m=+0.115923211 container create 9b153f7cce4d03746720f58753eac4cd1710503574587d0cd928404f44886e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-22T06:50:08.627 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.514096828 +0000 UTC m=+0.343239692 container init 9b153f7cce4d03746720f58753eac4cd1710503574587d0cd928404f44886e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:50:08.627 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.517041678 +0000 UTC m=+0.346184533 container start 9b153f7cce4d03746720f58753eac4cd1710503574587d0cd928404f44886e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:50:08.627 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.628800901 +0000 UTC m=+0.457943767 container attach 9b153f7cce4d03746720f58753eac4cd1710503574587d0cd928404f44886e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-22T06:50:08.989 INFO:journalctl@ceph.osd.11.smithi177.stdout:Apr 22 06:50:08 smithi177 podman[69273]: 2024-04-22 06:50:08.813645182 +0000 UTC m=+0.642788041 container died 9b153f7cce4d03746720f58753eac4cd1710503574587d0cd928404f44886e0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T06:50:09.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@osd.11.service' 2024-04-22T06:50:09.331 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:09.331 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-22T06:50:09.332 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-22T06:50:09.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a 2024-04-22T06:50:09.661 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:09 smithi105 systemd[1]: Stopping Ceph mds.a for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:09.918 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: 2024-04-22T06:50:09.662+0000 7fe653867700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:09.918 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:09 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a[88418]: 2024-04-22T06:50:09.662+0000 7fe653867700 -1 mds.a *** got signal Terminated *** 2024-04-22T06:50:13.369 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:13 smithi105 podman[92671]: 2024-04-22 06:50:13.116025194 +0000 UTC m=+3.587892209 container died 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:13.628 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:13 smithi105 podman[92671]: 2024-04-22 06:50:13.454392526 +0000 UTC m=+3.926259538 container cleanup 901e77493d1ad353734f29a543729d4ab122283846b000325cf47f2c8133aa98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-22T06:50:13.628 INFO:journalctl@ceph.mds.a.smithi105.stdout:Apr 22 06:50:13 smithi105 bash[92671]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-a 2024-04-22T06:50:13.752 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.a.service' 2024-04-22T06:50:13.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:13.789 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-22T06:50:13.789 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-22T06:50:13.789 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.d 2024-04-22T06:50:13.917 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:13 smithi105 systemd[1]: Stopping Ceph mds.d for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:14.418 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d[69266]: 2024-04-22T06:50:14.115+0000 7fec7cbaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:14.418 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:14 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d[69266]: 2024-04-22T06:50:14.115+0000 7fec7cbaf700 -1 mds.d *** got signal Terminated *** 2024-04-22T06:50:18.156 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:17 smithi105 podman[92792]: 2024-04-22 06:50:17.888027346 +0000 UTC m=+3.881365847 container died 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:18.418 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:18 smithi105 podman[92792]: 2024-04-22 06:50:18.241374296 +0000 UTC m=+4.234712788 container cleanup 5a6602ba459c5855abf1358d80144c3b9477b9518bebfbc780dec47903519600 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T06:50:18.418 INFO:journalctl@ceph.mds.d.smithi105.stdout:Apr 22 06:50:18 smithi105 bash[92792]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-d 2024-04-22T06:50:18.712 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.d.service' 2024-04-22T06:50:18.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:18.743 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-22T06:50:18.744 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-22T06:50:18.744 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g 2024-04-22T06:50:19.167 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:18 smithi105 systemd[1]: Stopping Ceph mds.g for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:19.168 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: 2024-04-22T06:50:19.044+0000 7fcc596ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:19.168 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:19 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g[87520]: 2024-04-22T06:50:19.044+0000 7fcc596ce700 -1 mds.g *** got signal Terminated *** 2024-04-22T06:50:21.376 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:21 smithi105 podman[92914]: 2024-04-22 06:50:21.114353976 +0000 UTC m=+2.162118793 container died 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-22T06:50:21.667 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:21 smithi105 podman[92914]: 2024-04-22 06:50:21.478001764 +0000 UTC m=+2.525766587 container cleanup 26bc788802514115c87e41570037b80af0eb9702044eb8b70a37147ac03b9088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:21.668 INFO:journalctl@ceph.mds.g.smithi105.stdout:Apr 22 06:50:21 smithi105 bash[92914]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-g 2024-04-22T06:50:21.911 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.g.service' 2024-04-22T06:50:21.941 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:21.941 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-22T06:50:21.942 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-22T06:50:21.942 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.j 2024-04-22T06:50:22.254 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:22 smithi105 systemd[1]: Stopping Ceph mds.j for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:22.667 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j[68879]: 2024-04-22T06:50:22.256+0000 7feedf29a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:22.668 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:22 smithi105 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j[68879]: 2024-04-22T06:50:22.256+0000 7feedf29a700 -1 mds.j *** got signal Terminated *** 2024-04-22T06:50:26.538 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:26 smithi105 podman[93034]: 2024-04-22 06:50:26.270952772 +0000 UTC m=+4.116699851 container died b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-22T06:50:26.888 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:26 smithi105 podman[93034]: 2024-04-22 06:50:26.614970406 +0000 UTC m=+4.460717494 container cleanup b85b6585757775a705c13a6b1943a36752dc627570c78730ebd0452801b32624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-22T06:50:26.888 INFO:journalctl@ceph.mds.j.smithi105.stdout:Apr 22 06:50:26 smithi105 bash[93034]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-j 2024-04-22T06:50:27.025 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.j.service' 2024-04-22T06:50:27.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:27.055 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-22T06:50:27.055 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-22T06:50:27.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.b 2024-04-22T06:50:27.341 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:27 smithi112 systemd[1]: Stopping Ceph mds.b for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:27.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:50:27.341+0000 7efd5b1a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:27.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:27 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b[61053]: 2024-04-22T06:50:27.341+0000 7efd5b1a2700 -1 mds.b *** got signal Terminated *** 2024-04-22T06:50:29.399 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:29 smithi112 podman[68192]: 2024-04-22 06:50:29.128779564 +0000 UTC m=+1.884696469 container died 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-22T06:50:29.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:29 smithi112 podman[68192]: 2024-04-22 06:50:29.492307499 +0000 UTC m=+2.248224404 container cleanup 9d0f7a3d9909b6d47861abc0f72500e5a7d23abb1425b153f64bc969815b3080 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-22T06:50:29.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Apr 22 06:50:29 smithi112 bash[68192]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-b 2024-04-22T06:50:29.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.b.service' 2024-04-22T06:50:29.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:29.938 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-22T06:50:29.938 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-22T06:50:29.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e 2024-04-22T06:50:30.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:30 smithi112 systemd[1]: Stopping Ceph mds.e for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:30.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: 2024-04-22T06:50:30.245+0000 7f6b7e1ac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:30.458 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:30 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e[64439]: 2024-04-22T06:50:30.245+0000 7f6b7e1ac700 -1 mds.e *** got signal Terminated *** 2024-04-22T06:50:33.394 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:33 smithi112 podman[68324]: 2024-04-22 06:50:33.125865986 +0000 UTC m=+2.968939500 container died 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_BRANCH=HEAD) 2024-04-22T06:50:33.708 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:33 smithi112 podman[68324]: 2024-04-22 06:50:33.504754021 +0000 UTC m=+3.347827552 container cleanup 0d20ce08e886ad2d3976e7456f5debc4af0b15aa73fc455bb858026121831712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416) 2024-04-22T06:50:33.708 INFO:journalctl@ceph.mds.e.smithi112.stdout:Apr 22 06:50:33 smithi112 bash[68324]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-e 2024-04-22T06:50:33.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.e.service' 2024-04-22T06:50:33.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:33.918 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-22T06:50:33.918 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-22T06:50:33.918 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h 2024-04-22T06:50:34.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:33 smithi112 systemd[1]: Stopping Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:34.458 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: 2024-04-22T06:50:34.207+0000 7fac7f794700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:34.458 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:34 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h[64859]: 2024-04-22T06:50:34.207+0000 7fac7f794700 -1 mds.h *** got signal Terminated *** 2024-04-22T06:50:37.373 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 podman[68444]: 2024-04-22 06:50:37.119474157 +0000 UTC m=+2.989342582 container died 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-22T06:50:37.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 podman[68444]: 2024-04-22 06:50:37.450029648 +0000 UTC m=+3.319898079 container cleanup 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-22T06:50:37.632 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 bash[68444]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h 2024-04-22T06:50:37.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service' 2024-04-22T06:50:37.892 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 podman[68457]: 2024-04-22 06:50:37.634718383 +0000 UTC m=+0.514604248 container remove 355627198fb912db578303b59e34cd712154005dcee557b33bf83982adae3ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:37.892 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-22T06:50:37.892 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 systemd[1]: Stopped Ceph mds.h for 48924e24-0071-11ef-bc93-c7b262605968. 2024-04-22T06:50:37.892 INFO:journalctl@ceph.mds.h.smithi112.stdout:Apr 22 06:50:37 smithi112 systemd[1]: ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.h.service: Consumed 1.692s CPU time. 2024-04-22T06:50:37.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:37.906 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-22T06:50:37.906 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-22T06:50:37.906 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k 2024-04-22T06:50:38.208 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:37 smithi112 systemd[1]: Stopping Ceph mds.k for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:38.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: 2024-04-22T06:50:38.227+0000 7fc605123700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:38.708 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:38 smithi112 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k[64444]: 2024-04-22T06:50:38.227+0000 7fc605123700 -1 mds.k *** got signal Terminated *** 2024-04-22T06:50:41.458 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:41 smithi112 podman[68569]: 2024-04-22 06:50:41.118857539 +0000 UTC m=+2.989365972 container died 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:41.943 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:41 smithi112 podman[68569]: 2024-04-22 06:50:41.712138991 +0000 UTC m=+3.582647467 container cleanup 176315a62756b444ffa447c0abda164d4bfc0bd1af586421e53760d11b318d0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-22T06:50:41.944 INFO:journalctl@ceph.mds.k.smithi112.stdout:Apr 22 06:50:41 smithi112 bash[68569]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-k 2024-04-22T06:50:42.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.k.service' 2024-04-22T06:50:42.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:42.113 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-22T06:50:42.113 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-22T06:50:42.113 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c 2024-04-22T06:50:42.414 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:42 smithi177 systemd[1]: Stopping Ceph mds.c for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:42.671 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: 2024-04-22T06:50:42.415+0000 7f7c8bea6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:42.671 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:42 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c[66019]: 2024-04-22T06:50:42.415+0000 7f7c8bea6700 -1 mds.c *** got signal Terminated *** 2024-04-22T06:50:45.398 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:45 smithi177 podman[69377]: 2024-04-22 06:50:45.122674654 +0000 UTC m=+2.813007052 container died 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-22T06:50:45.698 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:45 smithi177 podman[69377]: 2024-04-22 06:50:45.491630228 +0000 UTC m=+3.181962676 container cleanup 58826bb7254758b32018e4e15e0659052d62f2d71241794dd76610ad7a4b7540 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-22T06:50:45.698 INFO:journalctl@ceph.mds.c.smithi177.stdout:Apr 22 06:50:45 smithi177 bash[69377]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-c 2024-04-22T06:50:45.839 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.c.service' 2024-04-22T06:50:45.875 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:45.875 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-22T06:50:45.875 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-22T06:50:45.875 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.f 2024-04-22T06:50:46.239 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:45 smithi177 systemd[1]: Stopping Ceph mds.f for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:46.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: 2024-04-22T06:50:46.186+0000 7f0f91151700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:46.240 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:46 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f[61298]: 2024-04-22T06:50:46.186+0000 7f0f91151700 -1 mds.f *** got signal Terminated *** 2024-04-22T06:50:49.433 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:49 smithi177 podman[69511]: 2024-04-22 06:50:49.128236717 +0000 UTC m=+3.038949810 container died b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-86f7587, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:49.434 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:49 smithi177 podman[69511]: 2024-04-22 06:50:49.436091919 +0000 UTC m=+3.346805011 container cleanup b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:49.739 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:49 smithi177 bash[69511]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f 2024-04-22T06:50:49.739 INFO:journalctl@ceph.mds.f.smithi177.stdout:Apr 22 06:50:49 smithi177 podman[69525]: 2024-04-22 06:50:49.637210096 +0000 UTC m=+0.509655576 container remove b63431b0a6a1d89139a8bb17943306c6a7150a48a2f85a6343df511b299c8d60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-86f7587, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:49.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.f.service' 2024-04-22T06:50:49.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:49.785 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-22T06:50:49.785 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-22T06:50:49.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i 2024-04-22T06:50:50.087 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:49 smithi177 systemd[1]: Stopping Ceph mds.i for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:50.489 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: 2024-04-22T06:50:50.089+0000 7efcc5dd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:50.489 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:50 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i[66015]: 2024-04-22T06:50:50.089+0000 7efcc5dd4700 -1 mds.i *** got signal Terminated *** 2024-04-22T06:50:53.395 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:53 smithi177 podman[69631]: 2024-04-22 06:50:53.123364946 +0000 UTC m=+3.123001188 container died be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-22T06:50:53.395 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:53 smithi177 podman[69631]: 2024-04-22 06:50:53.397453919 +0000 UTC m=+3.397090169 container cleanup be6d7c54aff331b9d2fb5e3aabaa19c14a4c99c9a04f47554bf8f22e802ec50c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-86f7587, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-22T06:50:53.654 INFO:journalctl@ceph.mds.i.smithi177.stdout:Apr 22 06:50:53 smithi177 bash[69631]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-i 2024-04-22T06:50:53.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.i.service' 2024-04-22T06:50:53.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:53.856 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-22T06:50:53.856 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-22T06:50:53.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l 2024-04-22T06:50:54.239 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:53 smithi177 systemd[1]: Stopping Ceph mds.l for 48924e24-0071-11ef-bc93-c7b262605968... 2024-04-22T06:50:54.239 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: 2024-04-22T06:50:54.150+0000 7f41a0aa6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T06:50:54.239 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:54 smithi177 ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l[65488]: 2024-04-22T06:50:54.150+0000 7f41a0aa6700 -1 mds.l *** got signal Terminated *** 2024-04-22T06:50:57.398 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:57 smithi177 podman[69754]: 2024-04-22 06:50:57.117768336 +0000 UTC m=+3.046460565 container died 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-86f7587, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-22T06:50:57.649 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:57 smithi177 podman[69754]: 2024-04-22 06:50:57.40080056 +0000 UTC m=+3.329492781 container cleanup 301bce91db78d104ab510d0d2d0e12c79a62d7181bf429bcaf87d7cf35073e8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3df98b53968d4f80a5e4a8eef98419a71c20df8a9ae66ad24bfd71b414b692d2, name=ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-86f7587, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-22T06:50:57.649 INFO:journalctl@ceph.mds.l.smithi177.stdout:Apr 22 06:50:57 smithi177 bash[69754]: ceph-48924e24-0071-11ef-bc93-c7b262605968-mds-l 2024-04-22T06:50:57.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-48924e24-0071-11ef-bc93-c7b262605968@mds.l.service' 2024-04-22T06:50:57.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T06:50:57.851 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-22T06:50:57.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:50:58.040 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:00.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:51:00.627 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:03.014 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T06:51:03.204 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:05.479 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:51:05.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:51:05.542 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T06:51:05.567 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T06:51:05.569 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi105/crash 2024-04-22T06:51:05.570 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:51:05.611 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:51:05.611 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:51:05.613 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi112/crash 2024-04-22T06:51:05.614 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:51:05.733 INFO:teuthology.orchestra.run.smithi112.stderr:tar: /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:51:05.733 INFO:teuthology.orchestra.run.smithi112.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:51:05.735 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi177/crash 2024-04-22T06:51:05.736 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash -- . 2024-04-22T06:51:05.805 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/48924e24-0071-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T06:51:05.805 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-04-22T06:51:05.806 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T06:51:05.807 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-22T06:51:05.840 INFO:tasks.cephadm:Compressing logs... 2024-04-22T06:51:05.840 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:51:05.884 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:51:05.886 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:51:05.906 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:51:05.907 INFO:teuthology.orchestra.run.smithi105.stderr:: No such file or directory 2024-04-22T06:51:05.907 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:51:05.908 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:51:05.908 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-22T06:51:05.908 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-22T06:51:05.908 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T06:51:05.909 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.g.log: /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T06:51:05.909 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T06:51:05.909 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log: -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T06:51:05.913 INFO:teuthology.orchestra.run.smithi177.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:51:05.914 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-22T06:51:05.914 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T06:51:05.914 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-22T06:51:05.914 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-22T06:51:05.914 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.4% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:51:05.915 INFO:teuthology.orchestra.run.smithi177.stderr: 90.6% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.x.log: 87.4% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr: 90.4% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.6% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.3% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:51:05.922 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.a.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:: No such file or directory 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.7.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T06:51:05.923 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.h.log: 87.4% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr: 90.5% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:51:05.924 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:51:05.925 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.e.log: 91.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T06:51:05.927 INFO:teuthology.orchestra.run.smithi177.stderr: 91.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-22T06:51:05.927 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T06:51:05.927 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-22T06:51:05.927 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-22T06:51:05.928 INFO:teuthology.orchestra.run.smithi177.stderr: 83.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T06:51:05.928 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T06:51:05.929 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-22T06:51:05.932 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T06:51:05.937 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-22T06:51:05.941 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-22T06:51:05.943 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.d.log: 91.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:51:05.944 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-22T06:51:05.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:51:05.946 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-22T06:51:05.964 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T06:51:05.982 INFO:teuthology.orchestra.run.smithi105.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-22T06:51:05.995 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-22T06:51:06.009 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-22T06:51:06.010 INFO:teuthology.orchestra.run.smithi105.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-22T06:51:06.021 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.k.log: 92.8% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-22T06:51:06.030 INFO:teuthology.orchestra.run.smithi112.stderr: 93.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-22T06:51:06.039 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-22T06:51:06.044 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-22T06:51:06.052 INFO:teuthology.orchestra.run.smithi177.stderr: 92.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-22T06:51:06.199 INFO:teuthology.orchestra.run.smithi177.stderr: 91.9% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T06:51:06.216 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-22T06:51:06.275 INFO:teuthology.orchestra.run.smithi105.stderr: 90.3% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T06:51:06.325 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T06:51:06.677 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T06:51:06.677 INFO:teuthology.orchestra.run.smithi177.stderr: 93.8% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-22T06:51:06.717 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-22T06:51:06.731 INFO:teuthology.orchestra.run.smithi105.stderr: 93.3% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T06:51:06.767 INFO:teuthology.orchestra.run.smithi105.stderr: 93.2% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-22T06:51:06.776 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T06:51:06.831 INFO:teuthology.orchestra.run.smithi105.stderr: 90.6% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T06:51:06.844 INFO:teuthology.orchestra.run.smithi112.stderr: 93.5% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T06:51:06.880 INFO:teuthology.orchestra.run.smithi177.stderr: 93.0% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T06:51:06.904 INFO:teuthology.orchestra.run.smithi105.stderr: 93.1% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T06:51:06.928 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-22T06:51:06.930 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-04-22T06:51:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m1.029s 2024-04-22T06:51:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m4.357s 2024-04-22T06:51:06.930 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.180s 2024-04-22T06:51:06.993 INFO:teuthology.orchestra.run.smithi177.stderr: 93.2% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T06:51:06.995 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-04-22T06:51:06.995 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m1.094s 2024-04-22T06:51:06.995 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m4.115s 2024-04-22T06:51:06.995 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.166s 2024-04-22T06:51:07.017 INFO:teuthology.orchestra.run.smithi105.stderr: 93.2% -- replaced with /var/log/ceph/48924e24-0071-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T06:51:07.019 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-04-22T06:51:07.020 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m1.123s 2024-04-22T06:51:07.020 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m5.057s 2024-04-22T06:51:07.020 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.198s 2024-04-22T06:51:07.020 INFO:tasks.cephadm:Archiving logs... 2024-04-22T06:51:07.020 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi105/log 2024-04-22T06:51:07.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T06:51:07.808 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi112/log 2024-04-22T06:51:07.809 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T06:51:08.442 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi177/log 2024-04-22T06:51:08.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T06:51:09.081 INFO:tasks.cephadm:Removing cluster... 2024-04-22T06:51:09.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force 2024-04-22T06:51:09.283 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:09.555 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force 2024-04-22T06:51:09.746 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:10.007 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 48924e24-0071-11ef-bc93-c7b262605968 --force 2024-04-22T06:51:10.197 INFO:teuthology.orchestra.run.smithi177.stdout:Deleting cluster with fsid: 48924e24-0071-11ef-bc93-c7b262605968 2024-04-22T06:51:10.457 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T06:51:10.457 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T06:51:10.474 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T06:51:10.489 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T06:51:10.504 INFO:tasks.cephadm:Teardown complete 2024-04-22T06:51:10.505 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T06:51:10.519 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T06:51:10.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T06:51:10.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T06:51:10.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T06:51:10.656 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T06:51:10.669 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T06:51:10.669 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:51:10.671 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:51:10.674 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:51:10.684 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:51:10.685 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:51:10.687 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:^- hv01.front.sepia.ceph.com 2 6 377 46 +1204us[+1221us] +/- 573ms 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -88us[ -71us] +/- 73ms 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +102us[ +119us] +/- 80ms 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:51:10.689 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:51:10.690 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-04-22T06:51:10.690 INFO:teuthology.orchestra.run.smithi112.stdout:^- hv01.front.sepia.ceph.com 2 6 377 46 +1211us[+1202us] +/- 573ms 2024-04-22T06:51:10.690 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -92us[ -100us] +/- 73ms 2024-04-22T06:51:10.690 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +107us[ +99us] +/- 80ms 2024-04-22T06:51:10.690 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:51:10.692 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:51:10.692 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-04-22T06:51:10.692 INFO:teuthology.orchestra.run.smithi177.stdout:^- hv01.front.sepia.ceph.com 2 6 377 111 +1214us[+1225us] +/- 572ms 2024-04-22T06:51:10.693 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 110 -90us[ -78us] +/- 72ms 2024-04-22T06:51:10.693 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +99us[ +99us] +/- 80ms 2024-04-22T06:51:10.693 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:51:10.693 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T06:51:10.704 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T06:51:10.705 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T06:51:10.715 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T06:51:10.807 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T06:51:10.871 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T06:51:10.934 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:51:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:10.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:51:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:10.988 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:51:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:11.015 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T06:51:11.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T06:51:11.037 INFO:teuthology.task.internal:Duration was 1836.991371 seconds 2024-04-22T06:51:11.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T06:51:11.053 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T06:51:11.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T06:51:11.056 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T06:51:11.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T06:51:11.089 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:51:11.092 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:51:11.094 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:51:11.492 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T06:51:11.492 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-04-22T06:51:11.492 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T06:51:11.514 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-04-22T06:51:11.514 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T06:51:11.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-04-22T06:51:11.559 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T06:51:11.579 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T06:51:11.579 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T06:51:11.581 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T06:51:11.601 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T06:51:11.644 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T06:51:11.644 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T06:51:11.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T06:51:11.666 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T06:51:12.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T06:51:12.053 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T06:51:12.053 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T06:51:12.076 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T06:51:12.100 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T06:51:12.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T06:51:12.136 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.142 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.157 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-04-22T06:51:12.162 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-04-22T06:51:12.186 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-04-22T06:51:12.222 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:12.239 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:12.253 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:51:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:51:12.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T06:51:12.289 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T06:51:12.290 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi105 2024-04-22T06:51:12.290 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T06:51:12.330 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi112 2024-04-22T06:51:12.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T06:51:12.364 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130/remote/smithi177 2024-04-22T06:51:12.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T06:51:12.401 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T06:51:12.402 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T06:51:12.404 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T06:51:12.407 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T06:51:12.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T06:51:12.466 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T06:51:12.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T06:51:12.480 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T06:51:12.480 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T06:51:12.483 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T06:51:12.485 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T06:51:12.495 INFO:teuthology.orchestra.run.smithi105.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 06:51 /home/ubuntu/cephtest 2024-04-22T06:51:12.497 INFO:teuthology.orchestra.run.smithi112.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 06:51 /home/ubuntu/cephtest 2024-04-22T06:51:12.506 INFO:teuthology.orchestra.run.smithi177.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 06:51 /home/ubuntu/cephtest 2024-04-22T06:51:12.507 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T06:51:12.520 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T06:51:12.576 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T06:51:12.612 DEBUG:teuthology.nuke:shortname: smithi105 2024-04-22T06:51:12.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:51:12.656 DEBUG:teuthology.nuke:shortname: smithi112 2024-04-22T06:51:12.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:51:12.679 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.830935', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPA11MmHfT2Cz2yueIY3/AZ3HnxXL90Bx+Xt2XD5gaIbkpiWrejI8JgWy7Lkcj5rKE+FV7D9iNyy3TLW99f1a4g='} 2024-04-22T06:51:12.701 DEBUG:teuthology.nuke:shortname: smithi177 2024-04-22T06:51:12.701 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:51:12.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.833474', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCVZEhP1y27H0IGqMvfG9b1BSfrpzUiVOVMTvUFxg1JgX0Ki8vRk2Y+fYFd6u6QtVTA6LDlpM0QLQpZq66ocqBQ='} 2024-04-22T06:51:12.761 INFO:teuthology.orchestra.console.smithi105:Power off 2024-04-22T06:51:12.761 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T06:51:12.773 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-22_05:52:53-fs-wip-lusov-quiescer-distro-default-smithi/7668130', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:14:55.832193', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/cWEzCQzEc16LopmxSMc5SWmlLThkApqEjNKRAO+/sy6QsDmmTiqBJiYPlCrd1ptCnj881sJuxwlAyLkQen74='} 2024-04-22T06:51:12.811 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-04-22T06:51:12.811 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:12.828 INFO:teuthology.orchestra.console.smithi112:Power off 2024-04-22T06:51:12.828 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T06:51:12.864 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-22T06:51:12.875 INFO:teuthology.orchestra.console.smithi177:Power off 2024-04-22T06:51:12.875 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T06:51:12.885 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2024-04-22T06:51:12.885 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:12.918 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-04-22T06:51:12.925 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-04-22T06:51:12.926 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:12.975 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-22T06:51:16.865 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:16.919 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:16.975 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-22T06:51:16.977 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:17.030 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-04-22T06:51:17.086 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-04-22T06:51:20.978 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:21.031 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:21.087 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:21.098 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-04-22T06:51:21.141 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2024-04-22T06:51:21.198 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-04-22T06:51:21.199 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-04-22T06:51:21.299 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-04-22T06:51:25.145 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:51:25.255 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2024-04-22T06:51:25.356 INFO:teuthology.orchestra.console.smithi112:Power off completed 2024-04-22T06:51:25.461 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} duration: 1836.9913709163666 failure_reason: 'error during scrub thrashing: Command failed on smithi105 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls''' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-22T06:51:25.462 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:51:25.558 INFO:teuthology.run:FAIL